JP5087101B2 - Program, information storage medium, and image generation system - Google Patents

Description

  The present invention relates to a program, an information storage medium, an image generation system, and the like.

  In recent years, game devices capable of performing game operations using a game controller with a built-in motion sensor have been gaining popularity in place of game controllers having conventional operation buttons and direction keys. According to the game device having such an operation interface, an intuitive operation input by the player is possible, and the game operation can be simplified. As a conventional technique of a game apparatus that enables such an intuitive interface, there is a technique disclosed in Patent Document 1, for example.

  However, in a game controller that incorporates such a motion sensor, it is possible to detect the motion of the player shaking hands, but it is difficult to detect the motion of the player grabbing an object.

  In addition, when the character corresponding to the player performs an action of reaching out to the opponent character, the action of the hand is simply reaching out to the opponent character or reaching out to the opponent character. It is not easy to determine whether the action is to grab a character.

  Further, after the player performs an action of grabbing, it becomes an issue how to determine the action that the player wants to perform following the action of grabbing. For example, after grabbing the opponent's character's right shoulder with one hand, the other hand may attack the opponent character with the other hand, or with the other hand, holding the opponent character's left shoulder. Therefore, how to determine the operation performed in conjunction with such a gripping operation becomes a problem.

JP 2009-89629 A

  According to some aspects of the present invention, it is possible to provide a program, an information storage medium, an image generation system, and the like that can generate an image by detecting the grabbing motion and the grabbing interlocking motion of the player.

  According to one embodiment of the present invention, an image information acquisition unit that acquires image information from an image sensor and whether or not one of the player's right hand and left hand is performing a grabbing operation based on the image information are determined. When it is determined that the gripping motion determination unit and the one hand are performing the gripping motion, the other hand different from the one hand is gripped following the gripping motion based on the image information. The present invention relates to an image generation system including a grasp interlocking motion determination unit that determines whether or not an interlocking operation is performed, and an image generation unit that generates an image based on the determination of the gripping motion and the grip interlocking motion. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  According to one aspect of the present invention, it is determined based on image information from the image sensor whether one hand of the player is performing a grabbing operation. When it is determined that the grasping operation is being performed, it is determined based on the image information whether the other hand of the player is performing the grasping interlocking operation, and an image corresponding to these determination results is generated. As described above, according to one aspect of the present invention, it is possible to determine the player's gripping motion and the subsequent gripping interlocking motion based on the image information from the image sensor. Accordingly, the movement of the player is intuitively reflected in the generated image, and an image that has never been possible can be generated. Further, since the player's operation is directly reflected in the image, the player's virtual reality can be improved. In addition, by determining the grasp interlocking operation following the grasping operation, it is possible to provide various games that have been difficult to realize so far, and it is possible to diversify the game development and the game progress.

  In the aspect of the invention, the grasping interlocking operation determining unit may perform any grasping interlocking operation among the plurality of first to Mth grasping interlocking operations (M is an integer of 2 or more). It may be determined based on the image information from the image sensor.

  In this way, it is possible to determine which of the first to Mth grabbing interlocking movements has been performed, and to generate an image based on the determination result, so that a variety of grabbing interlocking movement images can be generated. become.

  In the aspect of the invention, the grasping interlocking motion determination unit may determine whether the other hand is included in the first to Mth grasping interlocking operations based on information on an image area corresponding to the other hand part. It may be determined which of the grasping interlocking operations is performed.

  In this way, it is possible to determine which grabbing interlocking action the player is performing by effectively using the information of the image area corresponding to the other hand part.

  In the aspect of the invention, the grasping interlocking operation determining unit may determine whether the other hand is the first to Mth grasping interlocking operations based on information on the image area in a given period or given timing. It may be determined which one of the grabbing interlocking operations is being performed.

  If the determination is made using the information on the image area in a given period in this way, for example, when the information on the image area changes instantaneously, it is possible to prevent a situation in which the information is grasped and erroneously determined as an interlocking operation. . In addition, if the determination is made using the information of the image area at a given timing, for example, it is possible to determine at that timing whether or not the grabbing interlocking operation has succeeded. Can be provided.

  In the aspect of the invention, the grasping interlocking operation determination unit may perform any one of the first to Mth grasping interlocking operations based on the area of the image area. It may be determined whether or not.

  In this way, it is possible to realize which grabbing interlocking operation the player is performing by image processing for obtaining the area of the image area, and it is possible to realize determination processing of the grabbing interlocking operation by simple image processing.

  In one aspect of the present invention, the image information acquisition unit acquires color image information in which a color value is set at each pixel position as the image information, and the grasping interlocking operation determination unit includes the other hand. Even if it is determined based on the area of the image area in the color image information corresponding to a part which of the first to Mth grabbing interlocking operations the other hand is performing the grabbing interlocking operation. Good.

  In this way, it is possible to realize the determination process of the grasping interlocking operation by the image processing using the color image information obtained by the image sensor.

  In one aspect of the present invention, the image information acquisition unit acquires depth information in which a depth value is set at each pixel position as the image information, and the grasping interlocking operation determination unit corresponds to the other hand. On the basis of the area of the image area in the depth information to be determined, it may be determined which of the first to Mth grasp interlocking operations the other hand is performing the grasp interlocking operation.

  In this way, it is possible to realize the determination process of the grasping interlocking operation by the image processing using the depth information obtained by the image sensor.

  In the aspect of the invention, the grasping interlocking motion determination unit may use a determination threshold value for determining the area of the image region as a distance between the display unit on which the generated image is displayed and the player. You may change according to information.

  In this way, even when the distance between the display unit and the player changes, it is possible to realize a highly accurate grasping operation determination process.

  In the aspect of the invention, the grasping interlocking motion determination unit may determine whether the other hand is included in the first to Mth grasping interlocking operations based on image information of a recognition member worn by the player on the hand. It may be determined which of the grasping interlocking operations is performed.

  In this way, since it is possible to realize the determination process of the grasping movement using the image information of the recognition member, it is possible to improve the accuracy of the determination process and the like as compared with the method using only the image of the player's hand.

  In one aspect of the present invention, the grasping interlocking motion determination unit may determine which of the first to Mth grasping interlocking operations is performed by the other hand by shape recognition processing on the image information of the other hand part. It may be determined whether or not a grab-link operation is being performed.

  In this way, when the grasping operation is performed with one hand of the player, the shape recognition process for the image of the other hand is performed, so that the determination process of the player's grasping interlocking operation can be realized.

  According to another aspect of the present invention, the information acquisition unit includes an information acquisition unit that acquires skeleton information that identifies an action of the player that can be seen from the image sensor based on the image information from the image sensor (the computer functions as the information acquisition unit). And the grasping interlocking movement determining unit may determine which of the first to Mth grasping interlocking actions is being performed based on the skeleton information. Good.

  In this way, it becomes possible to determine which of the first to Mth grasp interlocking operations has been performed by the other hand by effectively utilizing the skeleton information.

  According to another aspect of the present invention, the information acquisition unit includes an information acquisition unit that acquires skeleton information that identifies an action of the player that can be seen from the image sensor based on the image information from the image sensor (the computer functions as the information acquisition unit). The grasping interlocking movement determination unit may determine whether or not the other hand is performing a grasping interlocking action based on the skeleton information.

  In this way, it is possible to realize the determination process of the player's grasp-linked operation by effectively utilizing the skeleton information.

  In one embodiment of the present invention, skeleton information that identifies the player's movement visible from the image sensor is acquired based on the image information from the image sensor, and a grabbing movement is performed based on the acquired skeleton information. Alternatively, it may include an information acquisition unit that sets a processing area for image processing for determining a grab interlocking operation (a computer may function as the information acquisition unit).

  By doing this, it is only necessary to perform image processing in the set processing region, so that the processing load of the image processing can be reduced, and the determination process of the grasping interlocking operation can be realized with a small processing load.

  In one aspect of the present invention, a character control unit that controls a character corresponding to the player is included (a computer is caused to function as the character control unit), and the image generation unit performs the grabbing operation and the grabbing interlocking operation by the player. If it is determined that the character is performing, an image may be generated in which the character performs a grabbing motion and a grabbing interlocking motion.

  In this way, when the player performs the grabbing motion and the grabbing interlocking motion, the character's grabbing motion and grabbing interlocking motion in the game space are performed accordingly, so that the virtual reality of the player can be improved.

  In the aspect of the invention, the grasping interlocking operation determining unit may determine whether the other hand is performing an attacking operation as the grasping interlocking operation based on the image information from the image sensor. Good.

  In this way, it is possible to determine a series of actions such as grasping the opponent character or the like with one hand and attacking with the other hand, and generate an image corresponding thereto.

  In one aspect of the present invention, an attack calculation that sets an attack calculation parameter for an attack operation by the other hand based on the image information from the image sensor and performs an attack calculation process based on the attack calculation parameter May be included (the computer may function as an attack calculation unit).

  In this way, not only the determination of the grasping operation and the grasping interlocking operation, but also the setting processing of the attack calculation parameter, the image information from the image sensor can be used effectively.

  Further, according to one aspect of the present invention, a grip preparation operation determination unit that determines whether or not the player is performing a grip preparation operation based on the image information (the computer functions as the grasp preparation operation determination unit), When it is determined that the grip preparation operation is being performed, the gripping motion determination unit performs image processing on an image corresponding to the hand of the player that is performing the grip preparation operation, and results of the image processing Based on the above, it may be determined whether or not the player is performing a grasping operation following the grasp preparation operation.

  In this way, the player's grasp preparation operation and the subsequent grasp operation can be determined based on the image information from the image sensor. Accordingly, the movement of the player is intuitively reflected in the generated image, and an image that has never been possible can be generated.

  Further, in one aspect of the present invention, the gripping motion determination unit performs the gripping motion by the player based on information of an image area corresponding to the hand portion of the player that is determined to be performing the gripping preparation motion. It may be determined whether or not.

  In this way, it is possible to determine whether or not the player is performing a grabbing operation by effectively using information on the image area corresponding to the hand portion that is performing the grabbing preparation operation.

  In the aspect of the invention, the gripping motion determination unit may determine whether the player is performing a gripping motion based on information on the image area in a given period or a given timing. Good.

  If the determination is made using the information on the image area in a given period in this way, for example, when the information on the image area changes instantaneously, it is possible to prevent a situation in which it is erroneously determined as a grabbing operation. In addition, if the determination is made using the information of the image area at a given timing, for example, it is possible to determine at that timing whether or not the grabbing operation has succeeded. Provision etc. become possible.

  In the aspect of the invention, the gripping motion determination unit may determine whether or not the player is performing a gripping motion based on the area of the image area.

  In this way, whether or not the player is performing a grabbing operation can be realized by image processing for obtaining the area of the image area, and the grasping motion determination processing can be realized by simple image processing.

  In the aspect of the invention, the grasping motion determination unit may use a threshold value for determining the area of the image region as distance information between the display unit on which the generated image is displayed and the player. You may change according to.

  In this way, even when the distance between the display unit and the player changes, it is possible to realize a gripping motion determination process with high accuracy.

  In the aspect of the invention, the gripping motion determination unit may determine whether or not the player is performing a gripping motion based on image information of a recognition member worn by the player on a hand.

  In this way, the determination processing of the player's grasping motion can be realized using the image information of the recognition member, so that the accuracy of the determination processing can be improved as compared with the method using only the player's hand image.

  In one aspect of the present invention, the grasping motion determination unit determines whether the player is performing a gripping motion by shape recognition processing for image information of the hand portion of the player that is determined to be performing the gripping preparation motion. It may be determined whether or not.

  In this way, for example, by detecting that the player's palm is in the closed state from the open state by the shape recognition processing, it is possible to realize the determination processing of the player's gripping motion.

1 is a configuration example of an image generation system according to the present embodiment. 2A and 2B are explanatory diagrams of a method for acquiring color image information and depth information using an image sensor. FIGS. 3A and 3B are explanatory diagrams of a method for determining the player's grasping preparation operation. 4A and 4B are explanatory diagrams of a method for determining a player's grasping motion. The example of the image produced | generated by this embodiment. FIG. 6A and FIG. 6B are explanatory diagrams of a method for determining a grab interlocking operation. FIG. 7A and FIG. 7B are explanatory diagrams of a method for determining a grab interlocking operation. FIG. 8A and FIG. 8B are examples of images generated by this embodiment. FIGS. 9A to 9C are explanatory diagrams of a method for determining a gripping motion or a gripping interlocking motion based on the area of the image area of the hand portion. FIGS. 10A to 10C are explanatory diagrams of a method of changing the determination threshold value according to the distance. FIG. 11A and FIG. 11B are explanatory diagrams of gloves that are recognition members. FIGS. 12A to 12D are explanatory diagrams of a determination method using a recognition member. FIG. 13A to FIG. 13C are explanatory diagrams about problems in the determination using the recognition member. 14A to 14C are explanatory diagrams of a determination method using a recognition member. Explanatory drawing of the method of calculating | requiring the skeleton information of a player based on depth information. FIG. 16A and FIG. 16B are explanatory diagrams of a method for determining a grab interlocking operation using skeleton information of a player. FIG. 17A and FIG. 17B are explanatory diagrams of a method for setting a processing area for image processing using skeleton information of a player. An example of table data of attack calculation parameters. The flowchart explaining the process of this embodiment. The flowchart explaining the process of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Configuration FIG. 1 shows an example of a block diagram of an image generation system (game device) of the present embodiment. Note that the configuration of the image generation system of the present embodiment is not limited to that shown in FIG. 1, and various modifications may be made such as omitting some of the components (each unit) or adding other components. .

  The operation unit 160 is for a player to input operation data, and functions thereof are direction instruction keys, operation buttons, analog sticks, levers, various sensors (such as an angular velocity sensor and an acceleration sensor), a microphone, or a touch panel type. This can be realized with a display.

  The operation unit 160 includes an image sensor realized by, for example, a color image sensor or a depth sensor. Note that the function of the operation unit 160 may be realized only by the image sensor.

  The storage unit 170 serves as a work area for the processing unit 100, the communication unit 196, and the like, and its function can be realized by a RAM (DRAM, VRAM) or the like. Then, the game program and game data necessary for executing the game program are held in the storage unit 170.

  An information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and functions thereof by an optical disk (CD, DVD), HDD (hard disk drive), memory (ROM, etc.), and the like. realizable. The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. That is, in the information storage medium 180, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit). Is memorized.

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by an LCD, an organic EL display, a CRT, a touch panel display, an HMD (head mounted display), or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The auxiliary storage device 194 (auxiliary memory, secondary memory) is a storage device used to supplement the capacity of the storage unit 170, and can be realized by a memory card such as an SD memory card or a multimedia card.

  The communication unit 196 communicates with the outside (for example, another image generation system, a server, or a host device) via a wired or wireless network, and functions as a communication ASIC, a communication processor, or the like. It can be realized by hardware and communication firmware.

  Note that a program (data) for causing a computer to function as each unit of the present embodiment is obtained from an information storage medium of a server (host device) via an information storage medium 180 (or storage unit 170, auxiliary storage) via a network and communication unit 196. May be distributed to the device 194). Use of an information storage medium by such a server (host device) can also be included in the scope of the present invention.

  The processing unit 100 (processor) performs game processing, image generation processing, sound generation processing, and the like based on operation data from the operation unit 160, a program, and the like. The processing unit 100 performs various processes using the storage unit 170 as a work area. The functions of the processing unit 100 can be realized by hardware such as various processors (CPU, GPU, etc.), ASIC (gate array, etc.), and programs.

  The processing unit 100 includes an image information acquisition unit 102, an information acquisition unit 104, a game calculation unit 106, a grasp preparation operation determination unit 107, a grasp movement determination unit 108, a grasp interlocking operation determination unit 109, an attack calculation unit 110, and an object space setting unit. 112, a character control unit 114, a virtual camera control unit 118, an image generation unit 120, and a sound generation unit 130. The information acquisition unit 104 includes a skeleton information acquisition unit 105, and the character control unit 114 includes a movement processing unit 115 and a motion processing unit 116. Various modifications may be made such as omitting some of these components or adding other components.

  The image information acquisition unit 102 acquires image information from the image sensor. For example, information on an image captured by the image sensor is stored in the image information storage unit 171 of the storage unit 170. Specifically, information on the color image captured by the color image sensor of the image sensor is stored in the color image information storage unit 172, and information on the depth image captured by the depth sensor of the image sensor is stored in the depth information storage unit 173. Is done. The image information acquisition unit 102 reads the image information from the image information storage unit 171 to acquire the image information.

  The information acquisition unit 104 acquires skeleton information based on the image information acquired by the image information acquisition unit 102. Then, the player's action is specified based on the acquired skeleton information. Further, based on the skeleton information, for example, a processing area for image processing is set.

  The game calculation unit 106 performs game calculation processing. Here, as a game calculation, a process for starting a game when a game start condition is satisfied, a process for advancing the game, a process for calculating a game result, or a process for ending a game when a game end condition is satisfied and so on.

  The grip preparation motion determination unit 107, the grip motion determination unit 108, and the grip interlocking motion determination unit 109 perform determination processing of the grip preparation motion, the grip motion, and the grip interlocking motion, respectively. The attack calculation unit 110 performs attack calculation processing for the character corresponding to the player. Details of these will be described later.

  The object space setting unit 112 performs an object space setting process in which a plurality of objects are arranged. For example, various objects (polygon, free-form surface or subdivision surface) representing display objects such as characters (people, animals, robots, cars, ships, airplanes, etc.), maps (terrain), buildings, courses (roads), trees, walls, etc. The object is configured to place and set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Arrange objects. Specifically, the object data storage unit 175 of the storage unit 170 stores object data that is data such as the position, rotation angle, moving speed, and moving direction of the object (part object) in association with the object number. . The object space setting unit 112 performs a process of updating the object data for each frame, for example.

  The character control unit 114 controls a character that moves (moves) in the object space. For example, the movement processing unit 115 included in the character control unit 114 performs a process of moving a character (model object, moving object). For example, the character is moved in the object space based on operation information input by the player through the operation unit 160, a program (movement algorithm), various data, and the like. Specifically, a simulation process for sequentially obtaining character movement information (position, rotation angle, speed, or acceleration) every frame (for example, 1/60 second) is performed. Note that a frame is a unit of time for performing a movement process, a motion process, and an image generation process.

  The motion processing unit 116 included in the character control unit 114 performs motion processing (motion reproduction, motion generation) that causes the character to perform motion (animation). This motion processing can be realized by reproducing the motion of the character based on the motion data stored in the motion data storage unit 176.

  Specifically, the motion data storage unit 176 stores the position or rotation angle of each bone (each part object constituting the character) constituting the skeleton of the character (model object) around the three axes of the child bone relative to the parent bone. Motion data including the rotation angle). The model data storage unit 177 stores model data of a model object representing a character. The motion processing unit 116 reads this motion data from the motion data storage unit 176 and moves each bone (part object) constituting the skeleton based on the read motion data (by deforming the skeleton shape). Play the character's motion.

  The virtual camera control unit 118 performs control processing of a virtual camera (viewpoint, reference virtual camera) for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. Specifically, processing for controlling the position (X, Y, Z) or rotation angle (rotation angle about the X, Y, Z axis) of the virtual camera (processing for controlling the viewpoint position, the line-of-sight direction or the angle of view) I do.

  For example, when a character is photographed from behind using a virtual camera, the position or rotation angle (the direction of the virtual camera) of the virtual camera is controlled so that the virtual camera follows changes in the position or rotation of the character. In this case, the virtual camera can be controlled based on information such as the character position, rotation angle, or speed obtained by the movement processing unit 115. Alternatively, the virtual camera may be controlled to rotate at a predetermined rotation angle or to move along a predetermined movement path. In this case, the virtual camera is controlled based on the virtual camera data for specifying the position (movement path) or rotation angle of the virtual camera.

  The image generation unit 120 performs drawing processing based on the results of various processing (game processing and simulation processing) performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. Based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vector, α value, etc.) are created. Then, based on this drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is converted into image information in units of pixels such as a drawing buffer 179 (frame buffer, work buffer, etc. Draw in a buffer that can be stored. Thereby, an image that can be seen from the virtual camera (given viewpoint) in the object space is generated. Note that the rendering process can be realized by a vertex shader process or a pixel shader process.

  Note that the image generation unit 120 may generate a so-called stereoscopic image. In this case, the left-eye virtual camera and the right-eye virtual camera are arranged and set using the position of the reference virtual camera and the inter-camera distance. The image generation unit 120 generates a left-eye image that can be seen from the left-eye virtual camera in the object space and a right-eye image that can be seen from the right-eye virtual camera in the object space. Then, using these left-eye image and right-eye image, stereoscopic vision may be realized by an eye separation spectacle method, a naked eye method using a lenticular lens, or the like.

  The sound generation unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates game sounds such as BGM, sound effects, or sounds, and outputs the game sounds to the sound output unit 192.

  In the present embodiment, when the image information acquisition unit 102 acquires image information from the image sensor, the grasping motion determination unit 108 grasps one of the player's right hand and left hand based on the acquired image information. It is determined whether or not.

  When the gripping motion determination unit 108 determines that the gripping motion is being performed, the gripping interlocking motion determination unit 109 determines that the other hand of the player performs the gripping motion based on the image information from the image sensor. It is determined whether or not the subsequent grasp interlocking operation is performed. For example, if it is determined that the left hand, which is one hand of the player, is performing a grabbing motion, the grab interlocking motion determination unit 109 determines whether the right hand, which is the other hand, is performing the grabbing motion. To do. When it is determined that the right hand, which is one hand of the player, is performing a grabbing motion, the grab interlocking motion determination unit 109 determines whether the left hand, which is the other hand, is performing the grabbing motion. To do.

  Then, the image generation unit 120 generates an image based on the determination result of the grasping operation and the grasp interlocking operation. For example, after a character corresponding to the player performs a grabbing motion, an image that performs a grabbing interlocking motion is generated.

  Here, the grabbing interlocking operation (grasping cooperative operation) is an operation performed in conjunction with (in cooperation with) the grabbing operation after the grabbing operation. For example, it is an operation that is scheduled to be performed following the grasping operation, and an operation that is performed in conjunction with the grasping operation is associated in advance. Examples of the grabbing interlocking operation include an attacking motion (spinning motion, throwing motion, etc.), a stroking motion, a striking motion, a checking motion, a holding motion, or a gripping motion with the other hand (grabbing motion with both hands).

  Further, the grasping interlocking operation determining unit 109 determines from the image sensor which one of the plurality of first to Mth grasping interlocking operations (M is an integer of 2 or more) is performing the grasping interlocking operation. The determination is made based on the image information. Here, the first to M-th grabbing interlocking operations include the above-described attacking operation, stroking operation, tapping operation, examining operation, holding operation, and further grasping operation with the other hand. For example, when it is determined that one hand (for example, the left hand) of the player is performing an action of grabbing the opponent character or the like, the grasping interlocking movement determination unit 109 determines that the other hand (for example, the right hand) performs these attack actions. It is determined which of the first to M-th grabbing interlocking operations, such as a stroking motion and a hitting motion, is being performed. Specifically, the grasp interlocking operation is determined based on the image information from the image sensor. If it is determined that the Kth grasp interlocking operation (1 ≦ K ≦ M) among the first to Mth grasp interlocking operations is performed, the image generating unit 120 determines that the character corresponding to the player is used. However, after the grasping operation by one hand, an image for performing the K-th grasp interlocking operation by the other hand is generated.

  In this case, based on the information of the image area corresponding to the other hand portion, the grasping interlocking operation determining unit 109 performs any grasping interlocking operation among the first to Mth grasping interlocking operations. Determine if you are going. Specifically, based on the area of the image area, it is determined which of the first to Mth grabbing interlocking operations is being performed. For example, when it is determined that the player's right hand is performing the grabbing interlocking operation, the area of the image area occupied by the right hand image is obtained, and the right hand is selected from any of the first to Mth grabbing interlocking operations. Determine whether the grab interlocking operation is performed. When it is determined that the player's left hand is performing a grabbing interlocking operation, the area of the image area occupied by the left hand image is obtained, and the left hand is selected from any of the first to Mth grabbing interlocking operations. Determine whether the grab interlocking operation is performed.

  When the image information acquisition unit 102 acquires the color image information as the image information, the grasping interlocking operation determination unit 109 determines that the other hand of the player is the first to first based on the area of the image area in the color image information. It is determined which of the Mth grab interlocking movements is being performed. For example, the area of the image area of the color of the other hand (skin color, etc.) is obtained, and it is determined whether the other hand is performing a grasping interlocking operation. Here, the color image information is image information in which a color value is set at each pixel position, for example. When the image information acquisition unit 102 acquires the depth information as the image information, the grab interlocking motion determination unit 109 determines that the other hand of the player is based on the area of the image area in the depth information. It may be determined whether any of the grab interlocking movements is being performed. For example, the area of the image region corresponding to the depth value of the other hand is obtained, and it is determined whether the other hand is performing a grasping interlocking operation. Here, the depth information is image information in which a depth value is set at each pixel position.

  In addition, the grab interlocking movement determination unit 109 may change a determination threshold value for determining the area of the image region according to distance information between the display unit 190 on which the generated image is displayed and the player. . For example, when it is determined that the distance between the display unit 190 and the player is close, the determination threshold value for determining the area of the image region is set to a large value. On the other hand, if it is determined that the distance between the display unit 190 and the player is long, the determination threshold for determining the area of the image region is set to a small value. In this way, the determination threshold can be set to an appropriate value according to the distance between the display unit 190 and the player. It should be noted that the distance information between the display unit 190 and the player may be the distance between the display unit 190 and the player itself, or may be information equivalent to the distance. For example, the determination threshold value may be changed using the depth value at the representative position of the player or the position of the player's hand as distance information between the display unit 190 and the player.

  Note that the information on the image region used when determining the grasping-linked operation is not limited to the area of the image region, and the grasping-linked operation may be determined based on information on the image region other than the area. For example, the grasp-linked operation may be determined based on the color image information itself or the depth information itself in the image area.

  Further, when determining the grasp interlocking operation, it is desirable to determine the grasping operation based on the information of the image area in a given period. For example, after it is determined that a grasping operation is being performed, the grasp-linked operation is determined using information on image areas for a plurality of frames. That is, when the information such as the area of the image area changes instantaneously, the information of the image area maintained only for a given period is not subjected to the determination processing for the information that changed instantaneously. Is the target of the judgment process. In this way, when the information of the image area changes instantaneously due to noise or the like, it is possible to prevent a situation in which the player erroneously determines that the grasping interlocking operation is performed even though the grasping interlocking operation is not performed. It becomes like this.

  Further, when determining the grasp interlocking operation, it is desirable to determine the grasp interlocking operation based on the information of the image area at a given timing. For example, the player is instructed to use the image or sound to determine when to perform the grabbing interlocking operation, and the determination of the grabbing interlocking operation is performed based on information on the image area at the instruction timing (given timing). . Then, if it is determined from the information of the image area at the instruction timing that the player has succeeded in performing the grabbing and interlocking operation at the instruction timing, a score or the like is given to the player. By doing in this way, it becomes possible to provide a game of an unprecedented type in fields, such as a music game, an action game, and a sports game.

  Further, the grasping interlocking operation determination unit 109 performs any grasping interlocking operation among the first to Mth grasping interlocking operations based on the image information of the recognition member worn by the player on the hand. It may be determined whether or not. For example, the player wears a glove or the like as will be described later as a recognition member. Then, based on the color image information of a recognition member such as a glove, the player's grasping operation is determined.

  In addition, the grasping interlocking operation determination unit 109 determines which one of the first to Mth grasping interlocking operations is performed by the shape recognition process for the image information of the other hand part. You may judge. For example, when it is determined by shape recognition processing that the other palm is in an open state (par state), it is determined that an attack operation with a palm is performed as a grasping interlocking operation. On the other hand, when it is determined that the other palm is in a closed state (goo state), it is determined that an attack operation using a punch is performed as a grasp interlocking operation. The shape recognition process in this case can be realized by preparing a template image representing the shape of the hand with the palm open or closed, and performing a matching process with this template image.

  Note that the determination process of the grasping interlocking operation may be realized based on motion vector information or speed information in the image information of the other hand. For example, the motion vector and speed of the other hand are detected, and which one of the first to M-th grabbing interlocking movements is performed based on the detected motion vector and speed. Judgment.

  In addition, the information acquisition unit 104 acquires skeleton information that identifies the operation of the player as seen from the image sensor based on the image information from the image sensor. The acquired skeleton information is stored in the skeleton information storage unit 174.

  Here, the skeleton information is information for specifying, for example, the action of the player as seen from the image sensor. Specifically, this skeleton information has a plurality of joint position information corresponding to a plurality of joints of the player, and each joint position information is constituted by three-dimensional coordinate information. What connects each joint becomes a bone, and a skeleton is formed by connecting a plurality of bones.

  Then, based on the player's skeleton information, the grab interlocking motion determination unit 109 determines whether or not the other hand of the player is performing the grab interlocking motion. Further, based on the skeleton information, it is determined which of the first to Mth grasp interlocking operations the other hand of the player is performing. Specifically, it is determined based on the joint position information of the skeleton information which one of the first to Mth grasp interlocking movements is being performed. For example, first to Mth reference skeleton information serving as templates representing skeleton information in the first to Mth grab interlocking operations are prepared. Then, a matching process between the skeleton information of the player detected from the image information and the first to Mth reference skeleton information is performed. Then, using the result of the evaluation function of the matching process, it is determined which of the first to Mth grabbing interlocking movements the player is performing the grabbing interlocking action. The matching process in this case may be performed over a plurality of frames, for example.

  Further, when such skeleton information is acquired, the information acquisition unit 104 sets a processing area for image processing for determining a gripping operation or a gripping interlocking operation based on the skeleton information. For example, when it is determined that the player's left hand is performing a grab preparation operation described later, the information acquisition unit 104 specifies the position of the left hand based on the skeleton information, and has a predetermined size including the specified position. An area (an area smaller than the screen) is set as a processing area. Then, the grasping movement determination unit 108 performs image processing on the image information (color image information, depth information) of the set processing area, and determines whether or not the left hand is performing a grasping movement. If it is determined that the player's right hand is performing a grab preparation operation, the information acquisition unit 104 identifies the position of the right hand based on the skeleton information, and processes an area of a predetermined size including the identified position. Set to area. Then, the grasping motion determination unit 108 performs image processing on the image information of the set processing area, and determines whether or not the right hand is performing the gripping motion.

  For example, when it is determined that one hand of the player is performing a grabbing operation, the information acquisition unit 104 specifies the position of the other hand based on the skeleton information, and includes a predetermined position including the specified position. Set the size area as the processing area. Then, the grab interlocking motion determination unit 109 performs image processing on the image information of the set processing area, and determines whether the other hand is performing the grab interlocking motion.

  The character control unit 114 controls a character corresponding to the player. Here, the character becomes an object that is actually displayed in the third person viewpoint, but becomes a virtual object that is not displayed on the display unit 190 in the first person viewpoint.

  Then, when it is determined that the player is performing a grabbing motion and a grabbing interlocking operation, the image generation unit 120 generates an image in which the character performs the grabbing motion and the grabbing interlocking motion. For example, an image is generated in which a character performs a grabbing motion and then performs a grabbing interlocking motion. A series of actions of the character in this case may be realized by performing motion reproduction using the skeleton information of the player acquired from the image information from the image sensor.

  Further, the grab interlocking motion determination unit 109 determines whether the other hand is performing an attack motion as the grab interlocking motion based on the image information from the image sensor. In this case, the attack calculation unit 110 sets attack calculation parameters (parameters such as attack power and stiffness time) of the attack action by the other hand based on the image information from the image sensor. Then, attack calculation processing is performed based on the attack calculation parameters. For example, the motion vector information and speed information of the other hand are acquired from image information and skeleton information, and the attack calculation parameters for the attack action by the other hand are set based on the motion vector information and speed information.

  Further, the grab preparation operation determination unit 107 determines whether or not the player is performing a grab preparation operation based on image information from the image sensor. Here, the grip preparation operation is a series of operations performed before the player's hand grasps, and is a premise of the grasp operation. Specifically, in order to grasp the opponent character or item with a hand, the player performs a preparatory operation such as extending an arm, bending, or bending a wrist.

  When the grip preparation motion determination unit 107 determines that the grip preparation motion is being performed, the grip motion determination unit 108 performs image processing on the image corresponding to the hand of the player performing the grip preparation motion. Do. Then, based on the result of the image processing, it is determined whether or not the player is performing a grasping operation following the grasp preparation operation.

  For example, when it is determined that the player's left hand (left arm) is performing a holding preparation operation (for example, an operation to extend or bend the left arm), an image corresponding to the left hand (for example, an image area image including the left hand) is displayed. Then, image processing is performed. Also, if it is determined that the player's right hand (right arm) is performing a holding preparation operation (for example, an operation to extend or bend the right arm), an image corresponding to the right hand (for example, an image area image containing the right hand) Image processing. Then, based on the result of the image processing, it is determined whether or not the player's left hand or right hand is performing a grabbing operation. Then, the image generation unit 120 generates an image based on the determination result of the grasping operation. For example, when it is determined that a grabbing motion is being performed, an image in which the character (character hand) corresponding to the player performs the grabbing motion is generated.

  In addition, the gripping motion determination unit 108 determines whether or not the player is performing a gripping motion based on the information on the image area corresponding to the hand portion of the player that is determined to be performing the gripping preparation motion. Specifically, based on the area of the image area, it is determined whether or not a grasping operation is being performed. For example, if it is determined that the player's left hand is performing a grab preparation operation, the area of the image area occupied by the image of the left hand is obtained to determine whether the left hand is performing a grabbing operation. If it is determined that the player's right hand is performing a grab preparation operation, the area of the image area occupied by the right hand image is obtained to determine whether the right hand is performing a grabbing operation.

  When the image information acquisition unit 102 acquires color image information as image information, the grasping motion determination unit 108 determines whether or not the player is performing a grabbing motion based on the area of the image area in the color image information. judge. For example, the area of the image area of the hand color (skin color, etc.) is obtained to determine whether or not the player is performing a grabbing operation. When the image information acquisition unit 102 acquires depth information as image information, the gripping motion determination unit 108 determines whether or not the player is performing a gripping motion based on the area of the image area in the depth information. May be. For example, the area of the image area corresponding to the depth value of the hand is obtained, and it is determined whether or not the player is performing a grabbing operation.

  In addition, the grasping motion determination unit 108 may change the determination threshold for determining the area of the image region according to distance information between the display unit 190 on which the generated image is displayed and the player.

  Note that the image area information used when determining the gripping motion is not limited to the area of the image region, and it is determined whether the gripping motion is being performed based on the information of the image region other than the area. May be. For example, the grasping operation may be determined based on the color image information itself or the depth information itself in the image area.

  Further, when determining the grabbing motion, it is desirable to determine the grabbing motion based on information on the image area in a given period. For example, after it is determined that the holding operation is being performed, it is determined whether or not the holding operation is being performed using information on image areas for a plurality of frames. In this way, it is possible to prevent a situation in which an erroneous determination of the grasping operation is performed due to noise or the like.

  Further, when determining the gripping motion, it is desirable to determine the gripping motion based on the information of the image area at a given timing. For example, the player is instructed to use the image or sound to determine the timing at which the grasping operation should be performed, and the grasping operation is determined based on the image area information at the instruction timing (given timing). Then, when it is determined from the information of the image area at the instruction timing that the player has succeeded in performing the grabbing operation at the instruction timing, a score is given to the player. By doing in this way, it becomes possible to provide an unprecedented type of game.

  In addition, the gripping motion determination unit 108 may determine whether or not the player is performing a gripping motion based on the image information of the recognition member that the player wears on the hand.

  In addition, the gripping motion determination unit 108 may determine whether or not the player is performing a gripping motion by performing shape recognition processing on the image information of the hand portion of the player that is determined to be performing the gripping preparation motion. .

  Further, the grab preparation operation determination unit 107 may determine whether or not the player is performing a grab preparation operation based on the player's skeleton information. For example, reference skeleton information serving as a template representing skeleton information in the grip preparation operation is prepared. Then, matching processing between the skeleton information of the player detected from the image information and the reference skeleton information is performed, and based on the result of the evaluation function of the matching processing, it is determined whether or not the player has performed the grip preparation operation.

2. Next, the method of this embodiment will be described in detail.

2.1 Determination of Grasp Movement and Grasp Interlocking Operation Generally, a game device performs a game operation by operating a button or lever of a game controller. There is also a game apparatus in which a motion controller (six-axis sensor) is provided in the game controller, and a player performs a game operation by moving the game controller itself.

  However, such a game device with an operation interface requires a game controller for the game operation, and cannot implement a game operation that directly reflects a gesture such as a player's hand movement. I can not improve one more.

  Therefore, in this embodiment, an operation interface that allows a player to input an operation based on image information captured by the image sensor is employed.

  For example, in FIG. 2A, an image sensor ISE realized by a depth sensor (infrared sensor or the like) or a color image sensor (RGB sensor such as a CCD or CMOS sensor) is installed at a position corresponding to the display unit 190. The image sensor ISE is installed so that, for example, its imaging direction (optical axis direction) is a direction from the display unit 190 toward the player PL, and color image information of the player PL viewed from the display unit 190 side, Image depth information. The image sensor ISE may be built in the display unit 190 or may be prepared as an external component.

  Then, using the image sensor ISE in FIG. 2A, color image information and depth information as shown in FIG. 2B are acquired. For example, with color image information, color information of the player PL and the surrounding landscape can be obtained. On the other hand, in the depth information, the depth values (depth values) of the player PL and the surrounding scenery are obtained as gray scale values, for example. For example, the color image information is image information in which a color value (RGB) is set at each pixel position, and the depth information is image information in which a depth value (depth value) is set at each pixel position. The image sensor ISE may be a sensor in which a depth sensor and a color image sensor are provided separately, or may be a sensor in which a depth sensor and a color image sensor are combined.

  Various methods can be assumed as publicly known depth information acquisition methods. For example, by projecting light such as infrared rays from an image sensor ISE (depth sensor), the depth information is obtained by detecting the reflection intensity of the projected light and the time (Time Of Flight) until the projected light returns, The shape of an object such as the player PL viewed from the position of the image sensor ISE is detected. Specifically, the depth information is expressed as grayscale data that becomes brighter for an object close to the image sensor ISE and darker for an object far from the image sensor ISE, for example. The depth information acquisition method can be variously modified. For example, an image sensor such as a CMOS sensor may be used to acquire depth information (distance information to the subject) simultaneously with color image information. . Further, the depth information may be acquired by a distance measuring sensor using ultrasonic waves or the like.

  In this embodiment, when it is determined that the player PL is performing a grip preparation operation based on image information from the image sensor ISE, image processing is performed on an image corresponding to the hand performing the grip preparation operation. Then, it is determined whether or not a grasping operation following the grasp preparation operation is being performed.

  For example, in FIG. 3 (A) and FIG. 3 (B), the player PL performs a grip preparation operation that gradually raises the left hand HDL in order to perform the grip operation. In this case, in the present embodiment, image information such as color image information and depth information as shown in FIG. 2B is acquired by the image sensor ISE. Then, based on the acquired image information, it is determined whether or not the player PL is performing a holding preparation operation for the holding operation.

  For example, based on color image information and depth information as shown in FIG. 2B, the operation of the player PL is detected to determine whether or not a grasp preparation operation is being performed. Specifically, as will be described later, the player PL's skeleton information is detected based on the depth information of FIG. 2 (B), the movement of the joint in the skeleton information is analyzed, and the player PL performs a grasping preparation operation. It is determined whether or not. For example, first to N-th reference skeleton information serving as templates corresponding to a plurality of first to N-th holding preparation operations considered to be performed by the player PL are prepared. Then, a matching process is performed between the skeleton information of the player PL obtained from the depth information and the first to Nth reference skeleton information. Then, for example, the value of the evaluation function for matching between the Lth reference skeleton information (1 ≦ L ≦ N) of the first to Nth reference skeleton information and the skeleton information of the player PL is equal to or less than a predetermined value. In this case, it is determined that matching has occurred. Then, it is determined that the player PL is performing the Lth grasp preparation operation corresponding to the Lth reference skeleton information.

  Note that the determination method of the grasp preparation operation based on the image information is not limited to the method using the skeleton information, and various known operation determination methods can be employed. For example, the matching process may be performed using the color image information or the depth information itself to determine whether or not the player PL is performing a grip preparation operation. Alternatively, for example, only changes in information on joint positions such as wrists and elbows in the skeleton information may be monitored to determine whether or not the player PL is performing a holding operation.

  If it is determined that the player PL is performing a grip preparation operation as shown in FIGS. 3A and 3B, an image near the hand performing the grip preparation operation is used. Perform image processing. For example, image processing is performed on color image information and depth information in an image area near the hand, and based on the result of the image processing, it is determined whether or not the player is performing a gripping operation following the gripping preparation operation. To do.

  For example, FIGS. 4A and 4B show examples of color image information (depth information) captured by the image sensor ISE. In FIG. 4A, the palm of the left hand HDL of the player PL is in an open state (par state). On the other hand, in FIG. 4B, the palm of the left hand HDL is in a closed state (goo state). As described above, when the palm changes from the open state to the closed state, it is determined that the player PL is performing the grasping operation following the grasp preparation operation.

  If it is determined that the gripping operation has been performed in this manner, for example, an image as shown in FIG. 5 is generated and displayed on the display unit 190. In FIG. 5, an image is generated in which the hand HLOB (hand object) of the character corresponding to the player PL is grasping the shoulder of the opponent character (target object TOB). That is, in the present embodiment, a character corresponding to the player PL is moved in the object space (game space) by the operation of the player PL. The operation of the player PL in this case is realized, for example, by detecting the operation of the player PL by the image sensor ISE. Then, after the player PL performs a grip preparation operation as shown in FIGS. 3 (A) and 3 (B) and then performs a gripping operation as shown in FIGS. 4 (A) and 4 (B), FIG. An image as shown in FIG. 5 is generated.

  FIG. 5 is an example of a first-person viewpoint image, but a third-person viewpoint image may be generated. In the third-person viewpoint image, the whole body of the character (or at least the head of the character) is displayed, whereas in the first-person viewpoint image, the whole body of the character (head) is not displayed. For example, as shown in FIG. Only a part of the character's hand or the like is displayed.

  In the present embodiment, when it is determined that one hand of the player is performing a grabbing operation by the method described with reference to FIGS. 4A and 4B, the image information from the image sensor ISE is included in the image information. Based on this, it is determined whether or not the other hand different from the one hand is performing a grasping interlocking operation following the grasping operation.

  For example, in FIG. 6 (A) and FIG. 6 (B), the player PL performs the gripping operation as shown in FIG. 4 (A) and FIG. 4 (B) with the left hand HDL which is one hand and then the other hand Grasping with the right hand HDR, which is the hand, is performed. Specifically, in FIG. 6A, after performing an action of grasping the opponent character with the left hand HDL, a grasp interlocking action of attacking the opponent character with the palm of the hand with the right hand HDR is performed. On the other hand, in FIG. 6B, after performing the action of grabbing the opponent character with the left hand HDL, the grabbing interlocking action of attacking the opponent character with the right hand HDR by punching is performed.

  In this case, in the present embodiment, image information such as color image information and depth information as shown in FIG. 2B is acquired by the image sensor ISE. Then, based on the acquired image information, it is determined whether or not the player PL is performing a grasp interlocking operation following the grasping operation.

  For example, based on color image information and depth information as shown in FIG. 2B, the operation of the player PL is detected, and it is determined whether or not a grab interlocking operation is being performed. Specifically, as described later, based on the depth information of FIG. 2B, the player PL's skeleton information is detected, the joint movement in the skeleton information is analyzed, and the player PL performs the grabbing and interlocking operation. It is determined whether or not. For example, first to M-th reference skeleton information serving as templates corresponding to a plurality of first to M-th grabbing interlocking movements considered to be performed by the player PL are prepared. Then, matching processing is performed between the skeleton information of the player PL obtained from the depth information and the first to Mth reference skeleton information. Then, for example, the value of the evaluation function for matching between the Kth reference skeleton information (1 ≦ K ≦ M) of the first to Mth reference skeleton information and the skeleton information of the player PL is equal to or less than a predetermined value. In this case, it is determined that matching has occurred. Then, it is determined that the player PL is performing the K-th grasp interlocking operation corresponding to the K-th reference skeleton information.

  In this way, it can be determined based on the image information from the image sensor ISE which of the plurality of first to Mth grabbing interlocking operations the other hand of the player is performing.

  In addition, the determination method of the grab interlocking motion based on the image information is not limited to the method using the skeleton information, and various known motion determination methods can be employed. For example, the matching process may be performed using the color image information or the depth information itself to determine whether or not the player PL is performing a grabbing and interlocking operation. Alternatively, for example, only changes in information on joint positions such as wrists and elbows in the skeleton information may be monitored to determine whether or not the player PL is performing a grabbing interlocking operation.

  Alternatively, image processing is performed on an image region corresponding to the other hand that is performing the grabbing interlocking operation, and it is determined whether the other hand is performing the grabbing interlocking operation. It may be determined which of the grab interlocking movements is being performed.

  For example, FIGS. 7A and 7B show examples of color image information (depth information) captured by the image sensor ISE.

  In FIG. 7A, the left hand HDL, which is one hand, performs an action of grasping the opponent character or the like. In this case, image processing is performed on the image region corresponding to the right hand HDR which is the other hand. In FIG. 7A, it is detected by image processing that the right hand HDR, which is the other hand, is in a state where the palm is open (par state). In this way, when the left hand HDL, which is one hand, performs a grabbing operation and the right hand HDR, which is the other hand, detects that the palm is open, by image processing, the player PL As shown in FIG. 6 (A), it is determined that a grab interlocking operation that attacks with a palm is performed.

  In FIG. 7B, the left hand HDL, which is one hand, performs an operation of grasping the opponent character or the like, and therefore performs image processing on the image area corresponding to the right hand HDR, which is the other hand. In FIG. 7B, it is detected by image processing that the right hand HDR, which is the other hand, is in a state where the palm is closed (goo state). As described above, when the image processing detects that the left hand HDL, which is one hand, is performing the grasping operation and the right hand HDR, which is the other hand, is in a state where the palm is closed, the player PL As shown in FIG. 6B, it is determined that a grab interlocking operation that attacks with a punch is performed.

  As described above, according to the method described in FIGS. 7A and 7B, it is determined whether or not the other hand of the player PL is performing a grabbing interlocking operation, or a plurality of first to first operations are performed. It can be determined which of the M grabbing interlocking operations is being performed. That is, it is possible to determine which of the gripping interlocking operations is being performed by the other hand among the gripping interlocking operation of the palm attack and the gripping interlocking operation of the punch attack (among the first to Mth grip interlocking operations).

  Then, when it is determined that the grasping and interlocking operation has been performed as described above, for example, images as illustrated in FIGS. 8A and 8B are generated and displayed on the display unit 190. In FIG. 8A, an image is generated in which the left hand HLOB of the character corresponding to the player PL grabs the opponent character's shoulder and the right hand HROB is making a palm attack against the opponent character. In FIG. 8B, an image is generated in which the character's left hand HLOB grabs the opponent character's shoulder and the right hand HROB performs a punch attack on the opponent character.

  According to the method of the present embodiment described above, it is possible to determine the player's gripping motion and the subsequent gripping motion based on the image information from the image sensor. Therefore, it is possible to detect whether or not the player has performed the grasping operation or the grasping-linked operation by detecting the grasping operation or the grasping-related operation of the player that is difficult to detect depending on the game operation using the game controller. Therefore, it is possible to realize a game in which the game progresses when the character corresponding to the player moves in the game space (object space) and grabs the opponent character or item. In addition, the player performs a grabbing motion corresponding to the player by performing a grabbing motion similar to that in the real world. Accordingly, the movement of the player is intuitively reflected in the game image, and a game image that has never been possible can be generated. Further, since the player's operation is directly reflected in the game image, the player's virtual reality can be improved.

  Also, by determining the grasp-linked operation following the grasp operation, various games that have been difficult to realize can be provided. For example, it is possible to hold the opponent character in the game space with one hand and perform a slap attack, a punch attack, or a throwing attack with the other hand. Alternatively, an item placed in the game space can be grasped with one hand, and the item can be hit, stroked, or examined with the other hand. In this case, according to the present embodiment, it is possible to determine what kind of content the grasping interlocking operation by the other hand is. That is, it is possible to determine whether the other hand's grabbing interlocking action is a palm attack, a punch attack, a throwing attack, a hitting action, a stroking action, or a check action. Therefore, it is possible to change the subsequent game progress and game development in accordance with the determination result, and an unprecedented game can be realized.

  Further, in this embodiment, when it is determined that one hand is performing a grasping operation, skeleton information about the other hand and image processing for the other hand are performed, and a grasping interlocking operation by the other hand is performed. Is determined. Therefore, it is possible to detect the player's grasping interlocking operation appropriately and reliably.

  That is, if only the movement of the right hand HDR in FIG. 7A and FIG. 7B is detected by image processing or skeleton information and the grasp interlock operation is determined, the operation that is not the grasp interlock operation is erroneously grasped. There is a risk that it is determined that the operation is linked. In addition, image processing for detecting a grab interlocking operation must be performed on, for example, an image of the entire screen, and the processing load increases. In particular, if such image processing for detecting the interlocking motion and the detection processing of the interlocking motion based on the skeleton information are executed in all the frames, the processing load becomes excessive.

  In this regard, in the present embodiment, description will be made with reference to FIGS. 7A and 7B on the condition that it is determined that the gripping operation as shown in FIGS. 4A and 4B has been performed. The image processing for detecting the grabbing interlocking motion and the detection processing of the grabbing interlocking motion based on the skeleton information are executed. Therefore, for example, it is not necessary to execute the image processing for detecting the grabbing interlocking motion or the detection processing of the grabbing interlocking motion based on the skeleton information in all the frames, so that the processing load can be reduced. Further, if the gripping motion is determined before the determination of the gripping interlocking operation, it is possible to limit the range in which image processing for detecting the interlocking motion of the gripping is performed. For example, if it is determined that the left hand, which is one hand of the player, is performing a grabbing action, only the right hand, which is the other hand, may perform image processing for detecting the grabbing interlocking action. Therefore, the processing load can be reduced.

  In this embodiment, when it is determined that the player has performed the grip preparation operation, whether the player performs the grip operation by performing image processing on the image of the hand of the player who is performing the grip preparation operation. It is determined whether or not. Therefore, it is possible to detect the player's gripping motion appropriately and reliably.

  That is, as shown in FIGS. 4A and 4B, if the gripping motion is determined by detecting only that the palm of the player has changed from the open state to the closed state, There is a possibility that a non-negative motion is erroneously determined as a grabbing motion. In addition, it is necessary to perform image processing for gripping motion detection that detects that the palm of the player has changed from an open state to a closed state, for example, on an image of the entire screen, which increases the processing load. .

  In this regard, in the present embodiment, on the condition that it is determined that the grip preparation operation as shown in FIGS. 3 (A) and 3 (B) has been performed, in FIGS. 4 (A) and 4 (B). The image processing for detecting the grasping motion described is executed. Therefore, for example, it is not necessary to execute the image processing for detecting the grabbing motion in every frame, so that the processing load can be reduced. In addition, since the range of image processing for grasping motion detection can be limited, the processing load can be further reduced.

2.2 Determination Process Based on Area of Image Region Next, specific determination of the grasping operation in FIGS. 4A and 4B and the grasping interlocking operation in FIGS. 7A and 7B is performed. An example of the method will be described. For example, in the present embodiment, it is determined whether or not the player is performing a grasping operation based on the area of the image area of the player's hand portion that is determined to be performing the grasping preparation operation. Further, in this embodiment, when it is determined that one hand is performing a grabbing motion, whether or not the player is performing a grabbing interlocking motion based on the area of the image area of the other hand portion is determined. Judgment. In the following, the determination method of the grasping motion based on the area of the image area will be mainly described as an example, but this determination method can be similarly applied to the determination of the grasping interlocking operation.

  For example, FIG. 9A shows a state when the player's palm is opened (par state), and FIG. 9B shows a state when the player's palm is closed (goo state). . As shown in FIG. 9A and FIG. 9B, the area S of the image area of the player's hand is more closed when the palm of FIG. 9B is closed than when the palm of FIG. 9A is opened. The state is smaller. Therefore, when the area S of the image area of the hand is larger than the determination threshold value VT, it can be determined that the palm is in an open state, and when the area S is smaller than VT, the palm is closed. Can be determined.

  Accordingly, as shown in FIG. 9C, after it is determined that the player's grasping preparation operation has been performed, the area S of the image region of the corresponding hand decreases, and the area S is smaller than the determination threshold value VT. When it becomes, it can be determined that the player has performed a grabbing operation. On the other hand, if the area S of the image area of the hand does not become smaller than the determination threshold value VT even after a predetermined time has elapsed after it is determined that the grasp preparation operation has been performed, the grasp operation is performed. It can be determined that it was not missed.

  The area S may be the area of the image area in the color image information shown in FIG. 2B or the area of the image area in the depth information. For example, when color image information is used, a hue range corresponding to the hand color (skin color) is set, and it is determined whether or not the color at each pixel position in the color image information falls within this hue range. Then, the area S may be obtained by calculating the number of pixels whose hue value falls within the hue range. When depth information is used, a depth range corresponding to the depth value of the hand is set, and it is determined whether or not the depth value at each pixel position in the depth information falls within this depth range. Then, the area S may be obtained by calculating the number of pixels whose depth value falls within the depth range.

  Further, when the player PL approaches the display unit 190 as shown in FIG. 10A and when the player PL moves away from the display unit 190 as shown in FIG. The area S becomes a different area. For example, when the distance between the display unit 190 and the player PL is a short distance LDN as shown in FIG. 10A, the area S is a large value, and as shown in FIG. 10B, a long distance LDF is used. In some cases, the area S is a small value.

  Therefore, in the present embodiment, the determination threshold value VT for determining the area S of the image region is changed according to the distance information between the display unit 190 and the player PL. For example, as shown in FIG. 10C, when the distance L between the display unit 190 and the player PL is a short distance LDN, the determination threshold value VT is set to a large threshold value VTN. On the other hand, when the distance L is the long distance LDF, the determination threshold value VT is set to a small threshold value VTF. That is, the determination threshold value VT is reduced as the distance L increases.

  In this way, even when the player PL approaches the display unit 190 as shown in FIG. 10A, or when the player PL moves away as shown in FIG. 10B, an appropriate determination threshold value VT is set. By using this, the area S of the image region can be determined. Accordingly, it is possible to improve the accuracy of the determination process of the grasping operation and the grasping interlocking operation based on the area S.

  In FIG. 10C, the determination threshold value VT is set using the distance L between the display unit 190 and the player PL, but the present embodiment is not limited to this. For example, the determination threshold value VT may be set based on the character CH or the depth value of the hand of the character CH. In this case, the depth value corresponds to distance information indicating the distance between the display unit 190 and the player PL.

  Also, in order to improve the accuracy and accuracy of image processing, a player is provided with a recognition member for determination of a grasping motion or a grasp-linked motion, and the player performs a grasping motion based on image information of the recognition member. It is desirable to determine whether or not the scrambled operation is performed.

  For example, FIG. 11 (A) and FIG. 11 (B) show a glove ETH which is an example of such a recognition member. As shown in FIG. 11A, the glove ETH (recognition member in a broad sense) has its palm side set to red (first color in a broad sense). Also, as shown in FIG. 11B, the back side of the hand is set to blue (second color in a broad sense). Thus, by making the colors different between the palm and the back of the hand, it is possible to easily determine whether the player is pointing the palm toward the display unit 190 or the back of the hand.

  FIG. 12A shows a state where the palm on which the gloves ETH are worn is opened (par state). In this case, red, which is the color of the palm side of the glove ETH, occupies most of the image area of the hand. On the other hand, FIG. 12B shows a state where the palm is closed (goo state). In this case, red, which is the color on the palm side, occupies the first area of the image area of the hand, and blue, which is the color on the back side of the hand, occupies the second area.

  Therefore, in the present embodiment, as shown in FIG. 12C, after it is determined that the player's hand is performing a grab preparation operation, an image region of red (first color), which is the color on the palm side, is displayed. When the area SR decreases, it is determined that the player is performing a grabbing operation. Specifically, when the red color area SR is smaller than the determination threshold value VTR, it is determined that the player is performing a grabbing operation.

  Note that, as shown in FIG. 12D, when the area SB of the blue (second color) image region which is the color on the back of the hand is increased, it may be determined that the player is performing a grabbing operation. . Specifically, when the area SB of the blue color is larger than the determination threshold value VTB, it is determined that the player is performing a grabbing operation.

  Now, with the determination method described with reference to FIGS. 12A to 12D, there may be a problem described below.

  For example, consider the case where the player's hand rotates as shown in FIGS. 13 (A), 13 (B), and 13 (C). In FIG. 13A, the palm side faces the display unit 190 (image sensor), and in FIG. 13B thereafter, for example, the thumb side of the hand faces the display unit 190. Further, in FIG. 13C thereafter, the back side of the hand faces the display portion 190.

  When changing from FIG. 13A to FIG. 13B, the red area SR decreases. Therefore, if the red area SR becomes smaller than the determination threshold value VTR as shown in FIG. That is, even when the hand rotates as shown in FIGS. 13A to 13C, it is erroneously determined that the gripping operation shown in FIGS. 12A and 12B is performed. . The same erroneous determination occurs when an increase in the blue area SB is detected as shown in FIG.

  Therefore, in the present embodiment, as shown in FIGS. 14A and 14B, after it is determined that the player's hand is performing the holding preparation operation, the red (first color) image area SB is displayed. When the area is smaller than the first determination threshold value VTR and the area SB of the blue (second color) image area is smaller than the second determination threshold value VTB2, the player performs a grabbing operation. Determine that you are doing. In this way, the case where the hand rotates as shown in FIGS. 13A to 13C and the case where the gripping operation is performed as shown in FIGS. 12A and 12B are performed. , It becomes possible to distinguish and determine.

  That is, when the hand rotates as shown in FIGS. 13A to 13C, the red area SR decreases as shown in FIG. 14A, but as shown in FIG. 14C. The blue area SB also exceeds the second determination threshold value VTB2. Therefore, if it is detected that the red area SR decreases and the blue area SB does not exceed the second determination threshold value VTB2, the grasping operation of FIG. 12 (A) and FIG. 12 (B) is performed. It is possible to make a determination without erroneous determination.

  Note that the first determination threshold value VTR and the second determination threshold value VTB2 in FIGS. 14A and 14B are similar to those in the case of FIG. It is desirable to change according to the distance information between. That is, as the distance L between the display unit 190 and the player becomes longer, the first determination threshold value VTR and the second determination threshold value VTB2 are decreased.

  In this way, even when the distance between the display unit 190 and the player changes, the areas SR and SB of the image region can be determined using the appropriate determination threshold values VTR and VTB2. The accuracy of the grasping motion determination process can be improved.

  Note that the above-described determination method of the grasping operation can be applied to the grasping interlocking operation.

  For example, the determination process based on the area of the image region described with reference to FIGS. 9A to 9C can be applied not only to the gripping motion but also to the determination of the gripping interlocking motion. Specifically, if it is determined that one hand of the player is performing a grabbing operation, the other hand moves the first to first points based on the area of the image area corresponding to the other hand part. It is determined which of the M grabbing interlocking operations is being performed. For example, when the area S of the image region is large as shown in FIG. 9A, it is determined that the grasping interlocking operation of the palm attack shown in FIG. On the other hand, as shown in FIG. 9B, when the area S of the image region is small, it is determined that the punch attack grasping operation shown in FIG. 6B has been performed.

  In this case, the grasp interlocking operation may be determined based on the area of the image region in the color image information, or the grasp interlocking operation may be determined based on the area of the image region in the depth information.

  Further, also in the case of the determination of the grabbing interlocking operation, the generated image is displayed as the determination threshold value for determining the area of the image region as described with reference to FIGS. 10 (A) to 10 (C). It is desirable to change according to distance information between the display unit 190 and the player PL.

  In addition, the determination process based on the image information of the recognition member described with reference to FIGS. 11A to 14C can be applied not only to the gripping motion but also to the determination of the gripping interlocking motion. Specifically, based on the image information of the recognition member that the player wears on the hand, it is determined which of the first to M-th grasp interlocking operations is being performed by the other hand. For example, when the area SR of the red (first color) image region is large as shown in FIG. 12A, it is determined that the grasping interlocking operation of the palm attack shown in FIG. On the other hand, when the area SR of the red image region decreases or the area SB of the blue (second color) image region increases as shown in FIG. It is determined that the punching action shown in FIG.

  Moreover, the setting of the color of the glove which is a member for recognition is not limited to a setting like FIG. 11 (A) and FIG. 11 (B), A various deformation | transformation implementation is possible. Also, the player's grasping operation or grasping interlocking operation may be determined based on image information of a specific pattern such as a pattern drawn on a glove instead of the color itself. The recognition member is not limited to a glove. For example, a ring or the like worn on the player's finger may be used as a recognition member, and the player's grasping operation or grasping interlocking operation may be determined based on image information of the ring or the like.

  Further, the grasping operation or the grasping interlocking operation of the player may be determined by hand shape recognition processing. Specifically, it is determined whether or not the player is performing a grabbing operation or a grabbing interlocking operation by performing shape recognition processing on the image information of the part of the player's hand that has been determined to be performing the grabbing preparation operation. . For example, when the image-recognized hand shape changes from the shape of FIG. 9A to the shape of FIG. 9B, it is determined that the player is performing a grabbing operation. Alternatively, when the image-recognized hand shape is the shape shown in FIG. 9A, it is determined that the grasping interlocking operation of the palm attack shown in FIG. 6A has been performed, and FIG. In the case of the shape, it is determined that the punching attack interlocking operation shown in FIG. 6B has been performed. Various known image recognition processes can be employed as the shape recognition process in this case.

2.3 Grasp Interlocking Operation Now, as described above, in the present embodiment, it is determined based on image information from the image sensor ISE which of the plurality of grab interlocking operations is performed by the player PL. To do. That is, the player PL determines whether the grasp interlocking operation (slap attack) as shown in FIG. 6 (A) or the grasp interlocking operation (punch attack) as shown in FIG. 6 (B) is performed. The determination is based on image information from the image sensor ISE to be imaged.

  In this way, for example, when the grabbing interlocking operation shown in FIG. 6A is performed, the image corresponding to the grabbing interlocking operation is performed by the character so that the image shown in FIG. Can be generated. Further, when the grasping interlocking operation shown in FIG. 6B is performed, an image as shown in FIG. 8B can be generated by causing the character to perform an action corresponding to the grasping interlocking operation. .

  Accordingly, various grasping interlocking motion images are generated according to the movement of the player PL, and the variety of images generated can be increased. In addition, since it is possible to generate a grasp-linked motion image that is faithful to the movement of the player PL, it is possible to improve the virtual reality of the player PL.

  In this case, in the present embodiment, it may be determined based on the skeleton information of the player PL whether or not the player PL is performing a grabbing interlocking operation. For example, the skeleton of the player PL indicates which of the plurality of grip interlocking operations (first to Mth grip interlocking operations) as described in FIGS. 6A and 6B is being performed. Determine based on information.

  That is, in this embodiment, as shown in FIG. 15, skeleton information for specifying the operation of the player PL is acquired based on the image information such as the depth information of FIG. In FIG. 15, position information (three-dimensional coordinates) of the skeleton joints C0 to C19 is acquired as the skeleton information. These joints C0 to C10 correspond to the joints of the player PL reflected on the image sensor ISE. When the whole body of the player PL is not reflected on the image sensor ISE, skeleton information having position information on joints only for the reflected portion may be generated.

  For example, if the depth information in FIG. 2B is used, the three-dimensional shape of the player PL or the like viewed from the image sensor ISE side can be specified. Further, by combining color image information, a region of a part such as the face of the player PL can be specified by face image recognition or the like. Therefore, each part of the player PL is estimated based on the information on the three-dimensional shape and the joint position of each part is estimated. Then, based on the two-dimensional coordinates of the pixel position of the depth information corresponding to the estimated joint position and the depth information set to the pixel position, the three-dimensional coordinate information of the joint position of the skeleton is obtained, as shown in FIG. Get skeleton information.

  By using such skeleton information, the operation of the player PL gesture and the like can be specified in real time, and an unprecedented operation interface environment can be realized. Further, this skeleton information has a high affinity with the motion data of the character arranged in the object space. Therefore, it is possible to move the character in the object space using this skeleton information as, for example, motion data.

  In the present embodiment, the grasping-linked operation of the player PL is determined using the skeleton information shown in FIG.

  For example, FIG. 16A shows an example of skeleton information obtained by capturing an image of the player PL performing the grasping interlocking operation as shown in FIG. 6A with the image sensor ISE. On the other hand, FIG. 16B is an example of skeleton information obtained by capturing an image of the player PL that is performing the interlocking operation as shown in FIG. 6B with the image sensor ISE.

  As shown in FIGS. 16A and 16B, the player PL can grasp either of FIGS. 6A and 6B by using the information on the joint positions C0 to C19 of the skeleton information. It is possible to determine whether the interlocking operation has been performed.

  For example, by detecting a change in the position of the joints C7 and C6 in FIG. 16A corresponding to the right hand HDR in FIG. 6A, the grasping interlocking operation in FIG. Can be determined. In addition, by detecting a change in the positions of the joints C7 and C6 in FIG. 16B corresponding to the right hand HDR in FIG. 6B, the grasping interlocking operation in FIG. 6B in which a punch attack is performed with the right hand HDR. Can be determined.

  When skeleton information as shown in FIG. 16A is detected, an image as shown in FIG. 8A is generated as an image of the grasping interlocking operation in FIG. When skeleton information as shown in FIG. 16B is detected, an image as shown in FIG. 8B is generated as an image of the grasping movement operation in FIG. 6B.

  In this way, more accurate and appropriate determination processing can be realized by using the skeleton information obtained from the depth information to determine the grasp-linked operation. In addition, for example, the skeleton information used for reproducing the motion of the character can be effectively used to realize the determination process of the grab interlocking movement.

  Note that the operation determination process based on the skeleton information described above can be applied not only to the grasp interlocking operation, but also to the determination process of the grasp preparation operation described with reference to FIGS. 3A and 3A.

  In the present embodiment, the processing area for image processing may be set using the skeleton information acquired as shown in FIG.

  For example, assume that it is determined that the left hand, which is one hand of the player PL, is performing a grabbing action. In this case, as shown in FIG. 17A, an image area including the positions of joints C7 and C6 corresponding to the right hand which is the other hand of the player PL is set as a processing area RPC1 for image processing. Then, image processing for determination of the grasping interlocking operation is performed on the depth information and color image information in the set processing region RPC1. Specifically, in the processing region RPC1, image processing for obtaining the area of the image region as described with reference to FIGS. 9A to 14C is performed. Alternatively, in this processing area RPC1, hand shape recognition processing is performed to determine the grasp interlocking movement.

  In this way, it is not necessary to perform image processing in all regions, and image processing can be performed in the processing region RPC1 smaller than the screen size. Accordingly, the processing load of image processing can be reduced, and image processing for determination of the grasping interlocking operation can be executed with a small processing load.

  It should be noted that the image region setting process based on the skeleton information described above can be applied not only to the grasp interlocking movement determination process but also to the grasping movement determination process. Specifically, based on the skeleton information of the player PL, it is determined that the left hand of the player PL is performing a holding preparation operation. In this case, as shown in FIGS. 17A and 17B, the image area including the positions of the joints C11 and C10 corresponding to the left hand HDL of the player PL is set as the processing area RPC2 of the image processing. Then, image processing for determining the gripping operation is performed on the depth information and color image information in the set processing region RPC2. Specifically, in this processing region RPC2, image processing for determining the area of the image region as described with reference to FIGS. 9A to 14C is performed. Alternatively, a hand shape recognition process is performed in this processing area RPC2 to determine the gripping motion. In this way, it is possible to execute image processing for determination of the gripping operation with a small processing load.

2.4 Attack Calculation Processing In this embodiment, as described with reference to FIGS. 7A to 8B, based on the image information from the image sensor ISE, the other hand grasps the attack operation as a linked operation. It is determined whether it is going. For example, in FIG. 7 (A) and FIG. 8 (A), it is determined whether or not the other hand is performing an attack action with a palm. In FIG. 7 (B) and FIG. It is determined whether or not an attacking action is being performed.

  In this case, it is desirable to set the attack calculation parameter for the attack operation by the other hand based on the image information from the image sensor ISE and perform the attack calculation process based on the attack calculation parameter.

  For example, in the palm attack of FIGS. 7A and 8A, the speed and swing width of the right hand HDR are detected based on the image information of the right hand HDR that is the other hand. When the swing speed or swing width is large, the value of the attack power parameter, which is an attack calculation parameter, is increased. As a result, the damage amount of the opponent character due to the palm attack increases. On the other hand, when the swing speed or swing width is small, the value of the attack power parameter is decreased. As a result, the damage amount of the opponent character due to the palm attack is reduced.

  In this way, not only the determination of the grasping operation and the grasping interlocking operation but also the setting processing of the attack calculation parameters, the image information from the image sensor ISE can be used effectively. Further, the damage amount of the opponent character also increases in accordance with the speed of movement of the hand of the player PL and the amount of movement of the hand. Accordingly, the movement of the player is more effectively reflected on the game, and the virtual reality of the player can be improved.

  FIG. 18 shows an example of table data in which each attack calculation parameter is associated with each grab interlocking operation pattern. For example, an attack A is associated with the grab interlocking movement pattern PA, and the attack power parameter and the stiffness time parameter value of the attack A are associated with each other. Moreover, the attack-linked operation pattern PB is associated with the attack B, and the attack power parameter and the stiffness time parameter value of the attack B are associated with each other. Here, the stiffening time parameter is one of the parameters for attack calculation, and is a parameter representing a time for stiffening the opponent character when a player's attack hits. By preparing the table data as shown in FIG. 18, for example, it is possible to determine the grab interlocking operation pattern based on the skeleton information and execute the attack calculation using the attack calculation parameter associated with the determined pattern. become.

2.5 Detailed Processing Example Next, a detailed processing example of this embodiment will be described with reference to the flowcharts of FIGS. FIG. 18 is a flowchart showing the overall processing of this embodiment.

  First, as described in FIG. 2A and FIG. 2B, image information from the image sensor is acquired (step S1). Then, based on the acquired image information, it is determined whether or not one hand of the player is performing a grabbing motion as shown in FIGS. 4A and 4B (step S2).

  If it is determined that the gripping operation is being performed (step S3), image processing or the like is performed on the image corresponding to the other hand, and the other hand moves to FIG. 6 (A) to FIG. 7 (B). It is determined whether or not a grab interlocking operation as shown in FIG. If it is determined that the grabbing interlocking motion is being performed (step S5), as shown in FIGS. 8A and 8B, the character corresponding to the player performs the grabbing motion and the grabbing interlocking motion. An image to be performed is generated (step S6).

  FIG. 20 is a flowchart of processing for determining whether or not the player is performing a grabbing motion or a grabbing interlocking motion using the skeleton information of the player.

  First, as described in FIGS. 2A and 2B, image information such as color image information and depth information is acquired by the image sensor (step S11). Next, the player's skeleton information described with reference to FIG. 15 is obtained using the acquired depth information and the like (step S12). Then, based on the obtained skeleton information, it is determined whether one hand of the player is performing a grabbing motion (step S13). Specifically, based on the skeleton information, it is determined whether or not one hand of the player is performing a preparatory movement, and if it is determined that a preparatory movement is being performed, that one hand is grasped. It is determined whether or not an operation is being performed.

  If it is determined that the grasping operation is being performed (step S14), the position of the other hand is specified based on the skeleton information as described in FIGS. 17A and 17B. Then, an image area including the specified position is set as a processing area for image processing (step S15). Next, image processing is performed based on the image information in the set processing area, and it is determined whether or not the player is performing a grabbing interlocking operation (step S16).

  If it is determined that the grabbing interlocking motion is being performed (step S17), as shown in FIGS. 8A and 8B, the character corresponding to the player performs the grabbing motion and the grabbing interlocking motion. An image to be performed is generated (step S18).

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. In addition, the determination method of the player's grasping operation, the grasping interlocking operation and the grasping preparation operation based on the image information, the determination method based on the area of the image area, the acquisition method of the skeleton information, etc. are not limited to those described in this embodiment, Techniques equivalent to these are also included in the scope of the present invention. The present invention can be applied to various games. Further, the present invention is applied to various image generation systems such as a business game system, a home game system, a large attraction system in which a large number of players participate, a simulator, a multimedia terminal, a system board for generating a game image, and a mobile phone. it can.

ISE image sensor, PL player, SC screen, HDL left hand, HDR right hand,
HLOB character's left hand, HROB character's right hand,
TOB target object (other character),
RPC1, RPC2 treatment area, ETH gloves (recognition member),
100 processing unit, 102 image information acquisition unit, 104 information acquisition unit,
105 Skeleton information acquisition unit, 106 Game calculation unit, 107 Grasp preparation operation determination unit,
109 Grasping motion determination unit, 109 Grasping interlocking motion determination unit, 110 Attack motion determination unit,
112 object space setting unit, 114 character control unit, 115 movement processing unit,
116 motion processing unit, 118 virtual camera control unit, 120 image generation unit,
130 sound generation unit, 160 operation unit, 170 storage unit, 171 image information storage unit,
172 color image information storage unit, 173 depth information storage unit,
174 Skeleton information storage unit, 175 Object data storage unit,
176 motion data storage unit, 177 model data storage unit, 179 drawing buffer,
180 information storage medium, 190 display unit, 192 sound output unit, 194 auxiliary storage device,
196 Communication Department

Claims (25)

An image information acquisition unit for acquiring image information from the image sensor;
A gripping motion determination unit that determines whether one of the player's right hand and left hand is performing a gripping motion based on the image information;
If it is determined that the one hand is performing a grasping operation, based on the image information, is the other hand different from the one hand performing a grasp-linked operation following the grasping operation? A grab interlocking movement determination unit that determines whether or not,
As an image generation unit that generates an image based on the determination result of the grasping operation and the grasp interlocking operation,
A program characterized by causing a computer to function. In claim 1,
The grab interlocking movement determination unit is
It is determined, based on the image information from the image sensor, which one of a plurality of first to Mth grabbing interlocking movements (M is an integer of 2 or more) is performed by the other hand. The program characterized by doing. In claim 2,
The grab interlocking movement determination unit is
Based on the information of the image area corresponding to the other hand portion, it is determined which of the first to Mth grasp interlocking operations the other hand is performing the grasp interlocking operation. Program. In claim 3,
The grab interlocking movement determination unit is
Determining which of the first to Mth grabbing interlocking movements is performed by the other hand based on the information of the image area in a given period or given timing. A program characterized by In claim 3 or 4,
The grab interlocking movement determination unit is
A program for determining, based on the area of the image area, which of the first to Mth grasp interlocking operations is being performed by the other hand. In claim 5,
The image information acquisition unit
As the image information, color image information in which a color value is set at each pixel position is acquired,
The grab interlocking movement determination unit is
Based on the area of the image area in the color image information corresponding to the other hand portion, which one of the first to Mth grasp interlocking operations is being performed by the other hand The program characterized by determining. In claim 5,
The image information acquisition unit
As the image information, obtain depth information in which a depth value is set at each pixel position,
The grab interlocking movement determination unit is
Based on the area of the image area in the depth information corresponding to the other hand, it is determined which of the first to Mth grasp interlocking operations the other hand is performing the interlocking operation. A program characterized by that. In any of claims 5 to 7,
The grab interlocking movement determination unit is
A program for changing a threshold value for determining an area of the image area according to distance information between a display unit on which a generated image is displayed and the player. In claim 2,
The grab interlocking movement determination unit is
Based on the image information of the recognition member worn by the player on the hand, it is determined which of the first to Mth grasp interlocking operations the other hand is performing the grasp interlocking operation. Program. In claim 2,
The grab interlocking movement determination unit is
The shape recognition process for the image information of the other hand portion determines which of the first to Mth grasp interlocking operations is being performed by the other hand. program. In claim 2,
Based on the image information from the image sensor, as an information acquisition unit that acquires skeleton information that identifies the action of the player visible from the image sensor,
Make the computer work,
The grab interlocking movement determination unit is
A program for determining which of the first to Mth grab interlocking movements the other hand is performing based on the skeleton information. In claim 1,
Based on the image information from the image sensor, as an information acquisition unit that acquires skeleton information that identifies the action of the player visible from the image sensor,
Make the computer work,
The grab interlocking movement determination unit is
A program for determining whether or not the other hand is grasping and performing an interlocking operation based on the skeleton information. In claim 1,
Based on the image information from the image sensor, to acquire skeleton information that identifies the action of the player that can be seen from the image sensor, and to determine a gripping action or a gripping interlocking action based on the acquired skeleton information As an information acquisition unit that sets the image processing area
A program characterized by causing a computer to function. In any one of Claims 1 thru | or 13.
As a character control unit for controlling the character corresponding to the player,
Make the computer work,
The image generation unit
A program for generating an image in which the character performs a grabbing motion and a grabbing interlocking operation when it is determined that the player is performing a grabbing motion and a grabbing interlocking motion. In any one of Claims 1 thru | or 14.
The grab interlocking movement determination unit is
A program for determining, based on the image information from the image sensor, whether or not the other hand is performing an attack operation as a grabbing interlocking operation. In claim 15,
Based on the image information from the image sensor, set an attack calculation parameter of the attack action by the other hand, as an attack calculation unit that performs an attack calculation process based on the attack calculation parameter,
A program characterized by causing a computer to function. In any one of Claims 1 thru | or 16.
Based on the image information, as a grip preparation movement determination unit that determines whether or not the player is performing a grip preparation movement,
Make the computer work,
The gripping motion determination unit
When it is determined that a grasp preparation operation is being performed, image processing is performed on an image corresponding to the hand of the player that is performing the grasp preparation operation, and the player performs the image processing based on the result of the image processing. Determining whether or not a grasping operation following the grasp preparation operation is being performed. In claim 17,
The gripping motion determination unit
A program for determining whether or not the player is performing a grasping operation based on information on an image area corresponding to a hand portion of the player that has been determined to be performing a grasping operation. In claim 18,
The gripping motion determination unit
A program for determining whether or not the player is performing a grabbing motion based on information of the image area in a given period or given timing. In claim 18 or 19,
The gripping motion determination unit
A program for determining whether or not the player is performing a grabbing motion based on the area of the image area. In claim 20,
The gripping motion determination unit
A program for changing a threshold value for determining an area of the image area according to distance information between a display unit on which a generated image is displayed and the player. In claim 17,
The gripping motion determination unit
A program for determining whether or not the player is performing a grabbing operation based on image information of a recognition member worn on the hand of the player. In claim 17,
The gripping motion determination unit
A program characterized in that it is determined whether or not the player is performing a grasping operation by shape recognition processing on image information of the hand portion of the player that has been determined to be performing a grasping operation.   24. An information storage medium readable by a computer, wherein the program according to claim 1 is stored. An image information acquisition unit for acquiring image information from the image sensor;
A gripping motion determination unit that determines whether one of the player's right hand and left hand is performing a gripping motion based on the image information;
If it is determined that the one hand is performing a grasping operation, based on the image information, is the other hand different from the one hand performing a grasp-linked operation following the grasping operation? A grab interlocking movement determination unit that determines whether or not,
An image generation unit that generates an image based on a determination result of the grasping operation and the grasp interlocking operation;
An image generation system comprising: