JP2017138915A - Image generation system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generation system, a program and the like which can suppress generation of an improper character image due to incorrectness of skeleton recognition with skeleton information.SOLUTION: An image generation system includes: a subject information acquisition unit which acquires skeleton information on a subject on the basis of sensor information from a sensor; a character processing unit which performs processing for making a character move on the basis of the skeleton information on the subject; and an image generation unit which generates an image of the character. The character processing unit performs correction processing on the motion of the character on the basis of the skeleton information when it is determined that incorrectness occurs in skeleton recognition of the subject with the skeleton information.SELECTED DRAWING: Figure 24

Description

  The present invention relates to an image generation system, a program, and the like.

  2. Description of the Related Art Conventionally, a game image generation system that moves a character by motion reproduction using skeleton information is known. As a conventional technique of such an image generation system, for example, a technique disclosed in Patent Document 1 is known. There has also been proposed a system that captures skeleton information that identifies a player's movement in real time, and performs game processing using the captured skeleton information.

JP 2008-301987

  For example, if skeleton information representing the movement of the player is acquired in real time, a character that moves according to the movement of the player can be displayed by using the skeleton information. However, the player's skeleton recognition based on the skeleton information may be illegal. In this way, if the character is moved in a state where the skeletal recognition is illegal, the character may perform an unintended operation and an inappropriate image may be generated.

  According to some aspects of the present invention, it is possible to provide an image generation system, a program, and the like that can suppress generation of an inappropriate character image caused by skeletal recognition fraud based on skeleton information.

  One aspect of the present invention is a subject information acquisition unit that acquires skeleton information of a subject based on sensor information from a sensor, a character processing unit that performs a process of operating a character based on the skeleton information of the subject, An image generation unit configured to generate an image of the character, wherein the character processing unit is configured to perform an action of the character based on the skeleton information when it is determined that an illegality occurs in the skeleton recognition of the subject based on the skeleton information. The present invention relates to an image generation system that performs the correction process. The present invention also relates to a program that causes a computer to function as each of the above-described units, or a computer-readable information storage medium that stores the program.

  In one aspect of the present invention, skeleton information of a subject is acquired, and a process of moving a character based on the skeleton information is performed. That is, by using the skeleton information, a character motion corresponding to the movement of the subject is realized. If it is determined that the skeleton recognition of the subject based on the skeleton information is incorrect, the character motion correction process based on the skeleton information is executed. For example, if a character is moved based on skeleton information that makes skeletal recognition incorrect, there will be a situation in which the character moves inappropriately. In this regard, according to one aspect of the present invention, when it is determined that an illegality occurs in the skeleton recognition, the character motion correction process is performed, so that the occurrence of such a situation can be suppressed. Therefore, it is possible to provide an image generation system or the like that can suppress generation of an inappropriate character image due to skeletal recognition fraud based on skeleton information.

  In one aspect of the present invention, the character processing unit performs a motion blend of the skeleton motion based on the skeleton information and the reference motion of the character when it is determined that an illegality occurs in the skeleton recognition of the subject. The correction processing may be performed.

  Even if the skeletal recognition is fraudulent if such motion blending is performed, the reference motion is blended with the skeleton motion, resulting in an inappropriate character image caused by the skeletal recognition fraud. Generation can be suppressed.

  In one aspect of the present invention, the character processing unit may perform the motion blend only on bones that are determined to be fraudulent in the skeleton recognition of the subject.

  In this way, the processing load can be reduced compared to the case where motion blending is performed on all bones of the skeleton.

  In the aspect of the invention, the character processing unit may change the blend rate of the skeleton motion and the blend rate of the reference motion as time elapses.

  In this way, it is possible to realize a motion blend that gradually changes over time from a skeleton motion to a reference motion, and to generate a motion image of a character that looks more natural.

  In one aspect of the present invention, the character processing unit performs a motion replacement that replaces a skeleton motion based on the skeleton information with a given motion when it is determined that an illegality occurs in the skeleton recognition of the subject. The correction processing may be performed.

  If this kind of motion replacement is performed, even if skeleton recognition is fraudulent, replacing the skeleton motion with a given motion will generate inappropriate character images due to skeletal recognition fraud. It becomes possible to suppress.

  In one aspect of the present invention, the character processing unit may perform the correction process on a bone that is determined to be fraudulent in the skeleton recognition of the subject.

  In this way, it is possible to generate a proper character image by performing a bone correction process in which it is determined that fraud occurs in skeleton recognition.

  In one aspect of the present invention, the character processing unit acquires a right shoulder position and a left shoulder position of the skeleton from the skeleton information of the subject, and based on an angle of a line connecting the right shoulder position and the left shoulder position, When it is determined that an irregularity occurs in the bone of one of the left arm and the right arm of the subject, the correction processing may be performed on the bone of the one arm.

  In this way, based on the angle of the line connecting the right shoulder position and the left shoulder position, it is predicted that fraud will occur in the bone of one arm of the subject, and the correction process for the bone of the one arm is performed. It becomes possible to execute.

  In the aspect of the invention, the character processing unit may perform a motion blend of the skeleton motion based on the skeleton information and the reference motion of the character for the bone of the one arm that is determined to be illegal. .

  In this way, the correction process for one arm that is determined to be illegal can be realized by the motion blend of the skeleton motion and the reference motion.

  In the aspect of the present invention, the character processing unit may be configured to perform an operation on the foot bone when it is determined that an injustice occurs in the foot bone of the skeleton information based on the distance of the subject from the sensor. The correction process may be performed.

  In this way, it is possible to predict the occurrence of fraud in the foot bone based on the distance of the subject from the sensor and to perform correction processing on the foot bone.

  In the aspect of the invention, the character processing unit may replace the foot bone in the skeleton information with a given reference bone when it is determined that an injustice occurs in the foot bone.

  In this way, the motion of the character can be corrected to an appropriate motion by replacing the bone of the foot that is determined to be illegal in the skeleton recognition with the reference bone.

  According to another aspect of the present invention, the game processing unit includes a game processing unit that performs game processing (the computer is caused to function as the game processing unit), and the game processing unit is configured to determine that fraud occurs in the skeleton recognition of the subject. A game event prepared for the occurrence of the skeleton recognition fraud may be generated.

  In this way, when it is determined that fraud occurs in skeleton recognition, for example, a game event can be generated that avoids generation of an inappropriate character image due to fraud in skeleton recognition, for example. Become.

  According to another aspect of the present invention, the game processing unit includes a game processing unit that performs a game process (a computer is caused to function as the game processing unit), and the character processing unit performs the correction process of the character action according to a game situation. Correction processing may be performed.

  In this way, the correction process reflecting the game situation can be executed as the correction process for the occurrence of skeleton recognition fraud, and smooth game progress and the like can be realized.

  In the aspect of the invention, the character processing unit may perform the determination process for the character based on the action of the character after the correction process during the correction process.

  In this way, it is possible to execute an appropriate determination process for the character during the correction process.

1 is a configuration example of an image generation system according to the present embodiment. An application example of the image generation system of the present embodiment to a game machine for business use. 3A and 3B show application examples of the image generation system of the present embodiment to a home game device and a personal computer. 4A and 4B show examples of depth information and body index information. Explanatory drawing of skeleton information. Explanatory drawing of the image-synthesis method of a character image and a player's face image. FIG. 7A shows an example of a captured image of a player, and FIG. 7B shows an example of a combined image of a character and a face image. The example of the game image produced | generated by this embodiment. Explanatory drawing of the method of moving a character according to a player's motion. Explanatory drawing of the method of operating a character using skeleton information. Explanatory drawing of the rendering process of a model object. FIG. 12A to FIG. 12C are diagrams illustrating an example of the occurrence of a malfunction caused by skeletal recognition fraud based on skeleton information. FIG. 13A and FIG. 13B are explanatory diagrams of another example of the occurrence of a failure caused by skeleton recognition fraud. FIG. 14A and FIG. 14B are also explanatory diagrams of another example of the occurrence of a malfunction caused by skeleton recognition fraud. Explanatory drawing of the correction process of this embodiment by motion blend. Explanatory drawing of the correction process of this embodiment by motion replacement. FIG. 17A to FIG. 17D are explanatory diagrams of a method for determining the occurrence of skeleton recognition fraud using the angle of a line connecting both shoulders. Explanatory drawing of the motion blend method based on time passage. FIGS. 19A and 19B are explanatory diagrams of a technique for determining the occurrence of fraud in skeleton recognition using the distance from the sensor. The flowchart of the process which generate | occur | produces the game event for the time of fraud of skeleton recognition. FIG. 21A to FIG. 21C are explanatory diagrams of modified examples of the present embodiment. Explanatory drawing of the modification of this embodiment. FIG. 23A and FIG. 23B are also explanatory diagrams of a modified example of the present embodiment. The flowchart which shows the detailed process example of this embodiment. The flowchart which shows the detailed process example of this embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Image Generation System FIG. 1 shows a configuration example of an image generation system (image generation apparatus, game system, game apparatus) of the present embodiment. The image generation system includes a processing unit 100, an operation unit 160, a sensor 162, a storage unit 170, a display unit 190, a sound output unit 192, an I / F unit 194, a communication unit 196, and a printing unit 198. Note that the configuration of the image generation system of the present embodiment is not limited to that shown in FIG. 1, and various modifications may be made such as omitting some of the components (each unit) or adding other components.

  The processing unit 100 performs various processes such as an input process, an arithmetic process, and an output process based on operation information from the operation unit 160, sensor information from the sensor 162, a program, and the like.

  Each processing (each function) of this embodiment performed by each unit of the processing unit 100 can be realized by a processor (a processor including hardware). For example, each process of the present embodiment can be realized by a processor that operates based on information such as a program and a memory that stores information such as a program. In the processor, for example, the function of each unit may be realized by individual hardware, or the function of each unit may be realized by integrated hardware. The processor may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Processing Unit) can be used. The processor may be an ASIC hardware circuit. The memory (storage unit 170) may be a semiconductor memory such as SRAM or DRAM, or may be a register. Alternatively, it may be a magnetic storage device such as a hard disk device (HDD) or an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and the processing (function) of each unit of the processing unit 100 is realized by executing the instructions by the processor. The instruction here may be an instruction set constituting a program, or an instruction for instructing an operation to the hardware circuit of the processor.

  The processing unit 100 includes an input processing unit 102, an arithmetic processing unit 110, and an output processing unit 140. The input processing unit 102 performs various types of information input processing. For example, the input processing unit 102 performs a process of receiving player operation information input by the operation unit 160 as an input process. For example, a process of acquiring operation information detected by the operation unit 160 is performed. Further, the input processing unit 102 performs a process of acquiring sensor information from the sensor 162 as an input process. The input processing unit 102 performs a process of reading information from the storage unit 170 as an input process. For example, a process of reading information specified by the read command from the storage unit 170 is performed. Further, the input processing unit 102 performs processing for receiving information via the communication unit 196 as input processing. For example, a process of receiving information from an external device (another image generation system, server system, etc.) of the image generation system via a network is performed. The reception process is a process of instructing the communication unit 196 to receive information, acquiring information received by the communication unit 196, and writing the information in the storage unit 170.

  The arithmetic processing unit 110 performs various arithmetic processes. For example, the arithmetic processing unit 110 performs arithmetic processing such as game processing, object space setting processing, character processing, game results calculation processing, virtual camera control processing, image generation processing, or sound generation processing. The arithmetic processing unit 110 includes a game processing unit 111, an object space setting unit 112, a character processing unit 113, a subject information acquisition unit 114, a game result calculation unit 118, a virtual camera control unit 119, an image generation unit 120, and a sound generation unit 130. The print processing unit 132 is included.

  The game process includes a process for starting a game when a game start condition is satisfied, a process for advancing the game, or a process for ending a game when a game end condition is satisfied. This game processing is executed by the game processing unit 111 (game processing program module).

  The object space setting process is a process for arranging and setting a plurality of objects in the object space. This object space setting process is executed by the object space setting unit 112 (a program module for the object space setting process). For example, the object space setting unit 112 includes characters (people, robots, animals, monsters, airplanes, ships, fighters, tanks, battleships, cars, etc.), maps (terrain), buildings, courses (roads), trees that appear in the game. Various objects representing objects such as walls and water surfaces (objects composed of primitive surfaces such as polygons, free-form surfaces or subdivision surfaces) are arranged and set in the object space. In other words, the position and rotation angle of the object in the world coordinate system (synonymous with direction and direction) are determined, and the rotation angle (rotation angle around the X, Y, and Z axes) is determined at that position (X, Y, Z). Place the object. Specifically, the object information storage unit 172 of the storage unit 170 stores object information that is information such as the position, rotation angle, movement speed, and movement direction of the object (part object) in association with the object number. . The object space setting unit 112 performs processing for updating the object information for each frame, for example.

  Character processing (moving body calculation processing) is various calculation processing performed on a character (moving body). For example, the character processing is processing for moving a character (display object appearing in a game) in an object space (virtual three-dimensional space, three-dimensional game space) or processing for moving the character. This character processing is executed by the character processing unit 113 (character processing program module). For example, the character processing unit 113 uses a character (model) based on operation information input by the player through the operation unit 160, sensor information from the sensor 162, a program (movement / motion algorithm), various data (motion data), and the like. Control processing for moving the object) in the object space and moving the character (motion, animation) is performed. Specifically, a simulation process for sequentially obtaining character movement information (position, rotation angle, speed, or acceleration) and motion information (part object position or rotation angle) every frame (for example, 1/60 second). I do. The frame is a unit of time for performing character movement / motion processing (simulation processing) and image generation processing.

  The game score calculation process is a process of calculating the player's game score. For example, the game score calculation process is a process for calculating points and points that the player acquires in the game, and a process for calculating game results such as in-game currency, medals or tickets. This game score calculation processing is executed by the game score calculation unit 118 (program module for game score calculation processing).

  The virtual camera control process is a process for controlling a virtual camera (viewpoint, reference virtual camera) for generating an image that can be seen from a given (arbitrary) viewpoint in the object space. This virtual camera control process is executed by the virtual camera control unit 119 (virtual camera control process program module). Specifically, the virtual camera control unit 119 controls processing (viewpoint position, line-of-sight direction or image) for controlling the position (X, Y, Z) or rotation angle (rotation angle around the X, Y, Z axes) of the virtual camera. Process to control corners).

  The image generation processing is processing for generating an image (game image) displayed on the display unit 190 and an image printed by the printing unit 198, and may include various image composition processing and image effect processing. it can. The sound generation process is a process for generating a sound (game sound) such as BGM, sound effect, or sound output from the sound output unit 192, and can include various sound synthesis processes, sound effect processes, and the like. . These image generation processing and sound generation processing are executed by the image generation unit 120 and the sound generation unit 130 (program modules for image generation processing and sound generation processing).

  For example, the image generation unit 120 performs drawing processing based on the results of various processing (game processing and simulation processing) performed by the processing unit 100, thereby generating an image and outputting the image to the display unit 190. Specifically, geometric processing such as coordinate transformation (world coordinate transformation, camera coordinate transformation), clipping processing, perspective transformation, or light source processing is performed. Based on the processing result, drawing data (the position of the vertex of the primitive surface) Coordinates, texture coordinates, color data, normal vector, α value, etc.) are created. Based on the drawing data (primitive surface data), the object (one or a plurality of primitive surfaces) after perspective transformation (after geometry processing) is converted into image information in units of pixels such as a drawing buffer 178 (frame buffer, work buffer, etc.). Draw in a buffer that can be stored. Thereby, an image that is visible from a given viewpoint (virtual camera) in the object space is generated. Note that the drawing processing performed by the image generation unit 120 may be realized by vertex shader processing, pixel shader processing, or the like.

  The output processing unit 140 performs various types of information output processing. For example, the output processing unit 140 performs a process of writing information in the storage unit 170 as an output process. For example, a process of writing information specified by the write command to the storage unit 170 is performed. Further, the output processing unit 140 performs a process of outputting the generated image information to the display unit 190 and outputting the generated sound information to the sound output unit 192 as an output process. The output processing unit 140 performs a process of transmitting information via the communication unit 196 as an output process. For example, processing for transmitting information to an external device (another image generation system, server system, etc.) of the image generation system via a network is performed. The transmission process is a process of instructing the communication unit 196 to transmit information, or instructing the communication unit 196 to transmit information. Further, the output processing unit 140 performs processing for transferring an image to be printed on the print medium to the printing unit 198 as output processing.

  The operation unit 160 (operation device) is used by a player (user) to input operation information, and functions thereof are direction instruction keys, operation buttons, analog sticks, levers, various sensors (angular velocity sensor, acceleration sensor, etc.). ), A microphone or a touch panel display.

  The sensor 162 detects sensor information for acquiring subject information. The sensor 162 can include, for example, a color sensor 164 (color camera) and a depth sensor 166 (depth camera).

  The storage unit 170 (memory) serves as a work area for the processing unit 100, the communication unit 196, and the like, and its functions can be realized by a RAM, SSD, HDD, or the like. Then, the game program and game data necessary for executing the game program are held in the storage unit 170. The storage unit 170 includes an object information storage unit 172 and a drawing buffer 178.

  An information storage medium 180 (a computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), memory (ROM, etc.), etc. Can be realized. The processing unit 100 performs various processes of the present embodiment based on a program (data) stored in the information storage medium 180. A program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment on the information storage medium 180 (a program for causing the computer to execute processing of each unit). I can remember.

  The display unit 190 outputs an image generated according to the present embodiment, and its function can be realized by an LCD, an organic EL display, a CRT, an HMD, or the like. The sound output unit 192 outputs the sound generated by the present embodiment, and its function can be realized by a speaker, headphones, or the like.

  The I / F (interface) unit 194 performs interface processing with the portable information storage medium 195, and the function can be realized by an ASIC for I / F processing or the like. The portable information storage medium 195 is for a user to save various types of information, and is a storage device that retains storage of such information even when power is not supplied. The portable information storage medium 195 can be realized by an IC card (memory card), a USB memory, a magnetic card, or the like.

  The communication unit 196 communicates with an external device (another image generation system, server system, etc.) via a network, and functions thereof include hardware such as a communication ASIC or a communication processor, This can be realized by communication firmware.

  The printing unit 198 prints an image on a print medium such as print paper or sticker paper. The printing unit 198 can be realized by, for example, a printing head, a printing medium feeding mechanism, or the like. Specifically, the print processing unit 132 (print processing program module) of the processing unit 100 performs processing for selecting a print target image, and instructs the printing unit 198 to print the selected print target image. . Thereby, the printed matter on which the print target image is printed is paid out from a payout port 152 in FIG. 2 described later. The print target image is, for example, an image obtained by photographing the player's character during the game, or a commemorative photo image taken together with the player's character and another character.

  Note that a program (data) for causing a computer to function as each unit of the present embodiment is transferred from the information storage medium included in the server system (host device) to the information storage medium 180 (or storage unit 170) via the network and communication unit 196. You may distribute. Use of the information storage medium by such a server system can also be included in the scope of the present invention.

  2, 3 </ b> A, and 3 </ b> B are diagrams illustrating examples of hardware devices to which the image generation system of the present embodiment is applied.

  FIG. 2 is an example of an arcade game apparatus to which the image generation system of this embodiment is applied. This arcade game device is realized by an operation unit 160 realized by operation buttons and direction instruction buttons, a sensor 162 having a color sensor 164 and a depth sensor 166, a display unit 190 realized by an LCD, a CRT, and the like, and a speaker. A sound output unit 192, a coin insertion port 150, and a payout port 152 for printed matter such as photographs. The player PL performs various operations for playing the game while viewing the game image displayed on the display unit 190. The movement of the player PL (movement of a part such as a limb or movement of position) is detected by the sensor 162. A game process corresponding to the detection result is performed, and a game image based on the game process is displayed on the display unit 190.

  The hardware device to which the image generation system of the present embodiment is applied is not limited to the arcade game device as shown in FIG. 2, for example, a home game device shown in FIG. 3A or FIG. The present invention can be applied to various hardware devices such as a personal computer (information processing device) shown in FIG. In FIG. 3A, a sensor 162 (sensor device) having a color sensor 164 and a depth sensor 166 is connected to a main body device of a consumer game device, and the sensor 162 is disposed, for example, near a television which is a display unit. Then, by detecting the operation of the player by the sensor 162, the main device of the consumer game device executes various game processes and displays a game image on the television screen. In FIG. 3B, the sensor 162 is connected to the main unit of the personal computer, the operation of the player is detected by the sensor 162, and a game image based on the result of the game processing is displayed on a liquid crystal display as a display unit.

  The image generation system (image generation device, game device, game system) of the present embodiment includes a character processing unit 113, a subject information acquisition unit 114, and an image generation unit 120, as shown in FIG. The subject information acquisition unit 114 acquires skeleton information (movement information in a broad sense) of the subject based on the sensor information from the sensor 162. Based on this skeleton information, the movement of the subject is specified. The subject is, for example, the player PL shown in FIG. The subject may be an animal other than a human. The character processing unit 113 performs processing for moving the character based on the skeleton information, and the image generation unit 120 generates an image of the character. For example, the character is moved by motion reproduction based on skeleton information.

  The character processing unit 113 corrects the movement of the character based on the skeleton information when it is determined that the skeleton recognition of the subject based on the skeleton information is illegal.

  In this embodiment, based on sensor information from the sensor 162 that senses the subject, skeleton recognition of the subject is performed and skeleton information of the subject is acquired. However, if skeleton recognition based on this skeleton information is illegal. There is. When such skeletal recognition fraud occurs, the character's movement based on the skeleton information becomes inappropriate, and an inappropriate and unnatural image is generated.

  In this regard, in the present embodiment, when it is determined that an illegality occurs in the skeleton recognition of the subject, correction processing is performed on the character motion based on the skeleton information. For example, correction processing is performed to prevent the character's movement from becoming inappropriate. For example, correction processing is performed to reduce or eliminate the influence of skeletal recognition fraud on the character motion. In this way, even if an irregularity occurs in the skeleton recognition of the subject, the generated image becomes an inappropriate image by performing a correction process that prevents the character's movement from becoming inappropriate. Can be suppressed. Note that the case where it is determined that an illegality occurs in the skeleton recognition of the subject may be a case where an illegality actually occurs in the skeleton recognition, or it is predicted that an illegality occurs in the skeletal recognition (prediction, guess) It may be the case.

  The character processing unit 113 (motion processing unit) performs, for example, a motion blend of a skeleton motion based on skeleton information and a character's reference motion as a correction process when it is determined that the subject's skeleton recognition is illegal. For example, skeleton motion data based on skeleton information and a reference motion data prepared in advance are subjected to motion blending at a given blend rate. The reference motion data is stored in the motion data storage unit 174 of the storage unit 170. The reference motion is a motion that defines the reference posture (basic posture) of the character, for example. The motion data can include, for example, data such as the position (joint position) of the bone constituting the skeleton and the rotation angle of the bone.

  For example, suppose that the first motion is MA, the second motion is MB, the blend ratios α and β are each, and the motion after the motion blend is MT. In this case, the motion blend of the first motion MA and the second motion MB can be realized by a blend operation such as MT = α × MA + β × MB. Here, a relational expression such as α = 1−β holds. The data to be subjected to motion blending is, for example, data representing a bone position (joint position) and a bone rotation angle, and may be, for example, a bone local matrix.

  Further, the character processing unit 113 may perform motion blending only on bones that are determined to be fraudulent in skeleton recognition of the subject. For example, the motion blending with the reference bone included in the reference motion is performed only for the bones that are determined to be illegal in the skeleton recognition among the plurality of bones constituting the skeleton motion. By doing so, it is possible to reduce the processing load and processing data amount of motion blending.

  Further, the character processing unit 113 changes the blend rate of the skeleton motion and the blend rate of the reference motion, for example, according to the passage of time. For example, the blend rate of the skeleton motion is αS, and the blend rate of the reference motion is αF. In this case, for example, the skeleton motion blend rate αS is decreased and the reference motion blend rate αF is increased with the passage of time from the timing at which the skeleton recognition is determined to be fraudulent. By doing so, it is possible to realize a motion blend that gradually changes from a skeleton motion to a reference motion over time.

  Further, the character processing unit 113 may perform a motion replacement for replacing the skeleton motion based on the skeleton information with a given motion as the correction processing when it is determined that the skeleton recognition of the subject is illegal. That is, when it is determined that the skeleton recognition is illegal, the skeleton motion is replaced with a motion prepared in advance, and the motion processing of the character is performed. The motion data that is the replacement destination of the motion replacement is stored in the motion data storage unit 174. This replacement motion may be a fixed posture motion that does not change the character's posture, or may be a motion (time-series motion) that dynamically changes the character's posture.

  In addition, the character processing unit 113 performs a correction process on a bone that is determined to be illegal in skeleton recognition of the subject. For example, the correction process is not performed on all the bones of the character's skeleton, but the correction process is performed on the bones that are determined to be illegal in the skeleton recognition. By doing so, it is possible to reduce the processing load and processing data amount of the correction processing.

  In addition, the character processing unit 113 acquires the right shoulder position and the left shoulder position of the skeleton from the skeleton information of the subject. Then, based on the angle of the line connecting the right shoulder position and the left shoulder position, when it is determined that an irregularity occurs in the bone of one arm of the left arm or the right arm of the subject (a bone of a predetermined part in a broad sense), one arm Correction processing is performed on the bone. In other words, when it is expected that the skeleton recognition of a bone of one arm is incorrect based on the angle of the line connecting the right shoulder position and the left shoulder position, the bone of the one arm should not be an illegal bone. Correction processing is performed. Specifically, the character processing unit 113 performs a motion blend of the skeleton motion based on the skeleton information and the reference motion of the character with respect to the bone of one arm that is determined to be illegal. For example, motion blending is performed only on the bone of one arm that is determined to be illegal. Note that it is possible to perform a modification in which motion replacement is performed to replace the arm of the one bone with a given reference bone.

  Further, the character processing unit 113 corrects the foot bone when it is determined based on the distance of the subject from the sensor 162 that an injustice occurs in the foot bone of the skeleton information (a bone of a predetermined portion in a broad sense). Process. That is, based on the distance of the subject from the sensor 162, when it is determined that an irregularity occurs in the bone of the skeleton information, a correction process is performed to prevent the leg bone from becoming illegal. Specifically, the character processing unit 113 performs motion replacement that replaces the foot bone of the skeleton information with a given reference bone when it is determined that an injustice occurs in the foot bone. In this case, it is possible to perform a modification in which the motion blend process is performed.

  The image generation system according to the present embodiment includes a game processing unit 111 that performs game processing. The game processing unit 111 generates a game event prepared for the occurrence of skeleton recognition fraud when it is determined that the skeleton recognition of the subject is fraudulent. For example, when a correction process becomes necessary, a game event is generated that prompts an operation to avoid such a correction process. For example, when a player performs a given action, if an illegality occurs in the skeleton recognition, a game event that prevents the player from performing such an action is generated.

  In the present embodiment, the character processing unit 111 performs a correction process according to the game situation as a correction process for the character motion. For example, the content of the correction process is varied depending on the game situation. For example, when motion replacement is performed as the character motion correction processing for when skeleton recognition fraud occurs, motion replacement according to the game situation is performed. For example, when the character is performing an attack motion, a replacement motion for when a skeleton recognition fraud occurs is selected from the correction motions for attack. For example, the skeleton motion based on the skeleton information is replaced with a correction motion for attack. If the character is performing a defensive motion, a replacement motion for when a skeleton recognition fraud occurs is selected from the defensive correction motion. For example, a skeleton motion is replaced with a defensive correction motion. When a technique is selected during the game, a correction motion corresponding to the technique is selected as a replacement destination motion for when a skeleton recognition fraud occurs. For example, the skeleton motion is replaced with a correction motion corresponding to the technique.

  In addition, during the correction process, the character processing unit 113 may perform a determination process for the character based on the action of the character after the correction process. For example, based on the motion (motion) of the character after the correction process, a predetermined determination process for the progress of the game is performed. That is, when a predetermined determination process is required in the game process for the progress of the game, during the period of the correction process, based on the motion of the character after the correction process, not the character motion before the correction process, The determination process is performed. By doing so, the determination process can be executed in a state where the skeleton recognition is not illegal.

  The character is a model object that is composed of a plurality of objects and is arranged in the object space. For example, it is a three-dimensional model object composed of a plurality of three-dimensional objects (part objects). Each object is composed of, for example, a plurality of polygons (primitive surfaces in a broad sense).

  In this case, the image generation unit 120 performs rendering processing of the model object arranged at a position corresponding to the position of the subject (player or the like) in the object space, and generates a character image. For example, the position of the subject is specified based on the skeleton information (subject information) acquired by the subject information acquisition unit 114, and the model object that is the character is located at a position corresponding to the specified position of the subject in the object space. Be placed. Then, the rendering process of the model object is performed, and an image of the character that can be seen from the virtual camera is generated. In this case, the image generation unit 120 performs rendering processing of the model object based on the illumination model, and generates an image of the character. In this way, an image that is shaded by the light source of the illumination model can be generated as the character image.

  Further, the image generation unit 120 may generate a composite image by performing a combination process of a part image of a predetermined part of the subject included in the captured image of the subject and the character image. The captured image is, for example, an image (color image such as an RGB image) captured by a color sensor 164 (color camera) included in the sensor 162. The color sensor 164 and the depth sensor 166 may be provided separately. For example, a part image (for example, a face image) of a predetermined part of the subject is cut out from a captured image photographed by the color sensor 164, and a composite image is generated by combining the cut out part image with the character image. For example, a part image of the subject is synthesized with a part corresponding to a predetermined part of the character. For example, a face image that is a part image of a face that is a predetermined part is acquired by cutting out a face part from a captured image that shows the entire subject. The position of the face in the captured image can be specified based on the skeleton information (subject information) acquired by the subject information acquisition unit 114.

2. Next, the method of this embodiment will be described in detail. In the following, the case where the method of the present embodiment is applied to a game in which a character moves according to the movement of the player will be described. However, the method of the present embodiment is not limited to this, and various games (RPG games, music Game, battle game, communication game, robot game, card game, sports game, action game, etc.).

2.1 Description of Game First, an example of a game realized by the image generation system of this embodiment will be described. In the present embodiment, the movement of the player PL (subject in a broad sense) is detected by the sensor 162 in FIG. 2, and a game process that reflects the detected movement is performed to generate a game image.

  As shown in FIG. 2, the sensor 162 (image sensor) is installed so that the imaging direction (optical axis direction) faces the player PL (user, operator), for example. For example, the imaging direction of the sensor 162 is set in the depression direction with respect to the horizontal plane so that the entire child can be imaged. In FIG. 2, the sensor 162 is installed beside the display unit 190, but the installation position is not limited to this, and can be installed at an arbitrary position (for example, a lower part or an upper part of the display unit 190).

  The sensor 162 includes a color sensor 164 and a depth sensor 166. The color sensor 164 captures a color image (RGB image) and can be realized by a CMOS sensor, a CCD, or the like. The depth sensor 166 projects light such as infrared rays and detects depth of reflection information (depth information) by detecting the reflection intensity of the projected light and the time until the projected light returns. For example, the depth sensor 166 can be configured by an infrared projector that projects infrared rays and an infrared camera. For example, depth information is acquired by a TOF (Time Of Flight) method. The depth information is information in which a depth value (depth value) is set at each pixel position. In the depth information, the depth value (depth value) of the player and the surrounding landscape is set as, for example, a gray scale value.

  The sensor 162 may be a sensor in which the color sensor 164 and the depth sensor 166 are separately provided, or may be a sensor in which the color sensor 164 and the depth sensor 166 are combined. The depth sensor 166 may be a sensor other than the TOF method (for example, light coding). Various methods for obtaining depth information can be used. For example, depth information may be obtained by a distance measuring sensor using ultrasonic waves.

  4A and 4B are examples of depth information and body index information acquired based on the sensor 162, respectively. FIG. 4A conceptually shows depth information, and this depth information is information representing the distance of a subject such as a player viewed from the sensor 162. The body index information in FIG. 4B is information representing a person area. The position and shape of the player's body can be specified by this body index information.

  FIG. 5 is an example of skeleton information acquired based on sensor information from the sensor 162. In FIG. 5, position information (three-dimensional coordinates) of bones constituting the skeleton is acquired as position information of the joints (parts) C0 to C20 as the skeleton information. C0, C1, C2, C3, C4, C5, and C6 correspond to the waist, spine, center of shoulder, right shoulder, left shoulder, neck, and head, respectively. C7, C8, C9, C10, C11, and C12 correspond to the right elbow, right wrist, right hand, left elbow, left wrist, and left hand, respectively. C13, C14, C15, C16, C17, C18, C19, and C20 respectively correspond to the right hip, left hip, right knee, right heel, right foot, left knee, left heel, and left foot. Each bone constituting the skeleton corresponds to each part of the player reflected on the sensor 162. For example, the subject information acquisition unit 114 specifies the three-dimensional shape of the player based on sensor information (color image information and depth information) from the sensor 162. Then, using the information of the three-dimensional shape, the motion vector (optical flow) of the image, and the like, each part of the player is estimated, and the joint position of each part is estimated. Then, based on the two-dimensional coordinates of the pixel position in the depth information corresponding to the estimated joint position and the depth value set in the pixel position, the three-dimensional coordinate information of the position of the skeleton joint is obtained, and FIG. Acquire skeleton information as shown in. By using the acquired skeleton information, it is possible to specify the movement of the player (movement of a part such as a limb or movement of a position). This makes it possible to move the character according to the movement of the player. For example, when the player PL moves the limb in FIG. 2, the limb of the character (player character) displayed on the display unit 190 can be moved in conjunction with the movement. Further, when the player PL moves back and forth and right and left, the character can be moved back and forth and right and left in the object space in conjunction with the movement.

  In this embodiment, a composite image of a captured image obtained by photographing the player PL by the color sensor 164 of the sensor 162 and an image of a character (player character) is generated. Specifically, a composite image of a face image (subject image, part image in a broad sense) cut out from a captured image and a character image is generated.

  For example, in FIG. 6, a model object of a character CHP (costume) is prepared, and the face image IMF of the player PL cut out from the captured image is displayed on the polygon SCI (billboard polygon, screen). The model object of the character CHP is composed of objects of a plurality of parts other than the face (head), and is a character wearing a beautiful costume.

  In FIG. 6, a polygon SCI face image IMF is displayed on the extended line connecting the viewpoint position of the virtual camera VC and the face position of the character CHP. By setting the face image IMF of the virtual camera VC, the character CHP, and the polygon SCI in such an arrangement relationship, an image in which the face image IMF of the player is combined with the face portion of the character CHP can be generated.

  Note that the image synthesis method of the face image IMF for the character CHP is not limited to the method shown in FIG. For example, various image synthesis methods such as mapping the texture of the face image IMF (subject image, part image) to the object constituting the face (predetermined part) of the character CHP can be adopted.

  For example, FIG. 7A shows an example of a captured image of the player PL captured by the color sensor 164 of the sensor 162. A face image IMF of the player PL is cut out (trimmed) from the captured image and synthesized with the image of the character CHP by the image synthesis method of FIG. 6, thereby generating a composite image as shown in FIG. In FIG. 7B, the face part is the face image IMF of the player PL, and the part other than the face is an image of the character CHP that is a CG image. As a result, the player PL can play the game with the character CHP as his character, wearing a costume that is worn by the main characters of the fairy tales and anime, so that the fun of the game and the enthusiasm of the player can be improved. It can be improved.

  FIG. 8 is an example of a game image generated by the image generation system of the present embodiment and displayed on the display unit 190 of FIG. The character CHP combined with the face image IMF of the player PL has, for example, a magic wand ST in its hand. When the player PL in FIG. 2 moves the arm, the movement of the arm is detected by the sensor 162, and information on the skeleton in which the bone of the arm is moving is acquired as the skeleton information in FIG. As a result, the arm of the character CHP on the game image shown in FIG. 8 moves, and the enemy character EN1, EN2, EN3 can be hit and attacked with the magic wand ST. The player's game results are calculated based on the number of defeated enemies and the acquisition of bonus points.

  As described above, according to the method of the present embodiment, the player PL puts the costumes of the main characters of fairy tales and anime on the character CHP, which is his own character, and feels as if he is the main character, as shown in FIG. You will be able to play games like this. Therefore, it is possible to realize an interesting game that the player PL such as a child can enjoy.

2.2 Character Action According to Player Movement In this embodiment, the character action according to the player movement is realized by using skeleton information. For example, the character CHP displayed on the display unit 190 in FIG. 9 is a model object composed of a plurality of objects. For example, an image of the character CHP is generated with a three-dimensional CG object composed of a primitive surface such as a polygon. As shown in FIG. 9, an image in which the face image IMF of the player PL (subject) is displayed at the face position of the model object is generated as an image of the character CHP. Such a combined character image of the character CHP and the face image IMF can be generated by, for example, the image combining method described with reference to FIG.

  Although FIG. 9 shows a case where the face image IMF is synthesized with the character CHP, the present embodiment is not limited to this, and the synthesis of the face image IMF is not essential.

  In this embodiment, when the player PL (subject) moves as shown in FIG. 9, the character CHP displayed on the display unit 190 operates in conjunction with the movement of the player PL. For example, when the player PL moves to the right, the character CHP displayed on the display unit 190 also moves to the right accordingly. Similarly, when the player PL moves to the left, the character CHP displayed on the display unit 190 also moves to the left accordingly.

  Specifically, in this embodiment, the movement of the character CHP is controlled using the skeleton information described with reference to FIG. For example, the motion of the character CHP is reproduced by the motion data based on the skeleton information, and the character CHP is operated.

  FIG. 10 is a diagram conceptually illustrating the skeleton. The skeleton for moving the character CHP is a three-dimensional skeleton arranged and set at the position of the character CHP, and SK represents a projection of the skeleton on the screen SCS. As described with reference to FIG. 5, the skeleton is composed of a plurality of joints, and the position of each joint is represented by three-dimensional coordinates. The bone of the skeleton moves in conjunction with the movement of the player PL in FIG. When the bone of the skeleton moves, the part corresponding to the bone of the character CHP also moves in conjunction with this movement. For example, when the player PL moves his / her arm, the bone of the skeleton's arm moves and the arm of the character CHP moves accordingly. When the player PL moves his / her foot, the skeleton's foot bones move in conjunction with this, and the character CHP's feet also move. When the player PL moves his / her head, the skeleton's head bone moves and the character CHP's head moves accordingly. In FIG. 10, for example, it is assumed that the sensor 162 is disposed at the viewpoint position of the virtual camera VC, and the imaging direction of the sensor 162 is oriented in the line-of-sight direction of the virtual camera VC.

  Thus, in this embodiment, the character CHP operates in conjunction with the movement of the player PL. For this reason, it is possible to give the player a feeling as if the character CHP is his true alternation.

  In this embodiment, a model object composed of a plurality of objects is used as the character CHP. Then, an image of the character CHP is generated by performing rendering processing of the model object arranged at a position corresponding to the position of the player PL (subject) in the object space (three-dimensional virtual space). For example, in FIG. 11, the character CHP is arranged at a position corresponding to the position of the player PL in FIG. 9 in the object space. For example, when the player PL moves forward / backward / left / right, the character CHP also moves forward / backward / left / right in the object space.

  And the rendering process of the character CHP which is a model object is performed based on the illumination model using the light source LS as shown in FIG. Specifically, lighting processing (shading processing) based on the illumination model is performed. This lighting processing includes information on the light source LS (light source vector, light source color, brightness, light source type, etc.), a line-of-sight vector of the virtual camera VC, a normal vector of an object constituting the character CHP, and an object material (color). , Material) and the like. Illumination models include the Lumbard diffuse lighting model that considers only ambient light and diffuse light, the phone lighting model that considers specular light in addition to ambient light and diffuse light, and the Bling phone lighting model. is there.

  By performing the lighting process based on the illumination model in this way, it is possible to generate a real and high-quality image in which the character CHP is appropriately shaded by the light source LS. For example, if lighting processing is performed using a spotlight light source, an image of a character CHP (costume) that appears to be illuminated by the spotlight can be generated, and the virtual reality of the player can be improved.

2.3 Character Movement Correction Processing As described above, in this embodiment, the action of the character CHP according to the movement of the player PL is realized by using the skeleton information of the player PL acquired by the sensor 162. .

  However, since the skeleton information is acquired by sensing the player PL by the sensor 162, there are cases where the skeleton recognition based on the skeleton information is incorrect (error). For example, FIG. 12 (A) to FIG. 12 (C) are diagrams for explaining an example of the occurrence of a malfunction caused by skeletal recognition fraud based on skeleton information.

  In FIG. 12A, the player PL faces the sensor 162, and the arms AMR and AML are within the imaging range (sensing range) of the sensor 162. For this reason, the skeleton recognition about arm AMR and AML can be performed appropriately. In FIG. 12B, the player PL facing the sensor 162 is rotating clockwise. In this case, the arms AMR and AML are also in the imaging range of the sensor 162. For this reason, the skeleton recognition about arm AMR and AML can be performed appropriately.

  On the other hand, in FIG. 12C, the player PL is facing sideways with respect to the sensor 162. For this reason, the arm AML enters the imaging range, but the arm AMR does not enter the imaging range. That is, the arm AMR is hidden behind the body of the player PL and cannot be seen from the sensor 162. Therefore, the skeleton information acquired based on the sensor information from the sensor 162 in the state of FIG. 12C has a very high possibility that the skeleton recognition is incorrect (error). Therefore, if the character CHP is operated as shown in FIG. 9 using such illegal skeleton information, the character CHP moves in an incorrect manner, and the generated image is inappropriate and unnatural. It becomes an image. For example, an image that looks as if the bone has been broken is generated, such as bending the arm of the character CHP (an arm corresponding to the AMR) in a direction where there is no possibility.

  FIG. 13A to FIG. 14B are diagrams for explaining another example of the occurrence of a malfunction caused by skeleton recognition fraud based on skeleton information.

  In FIG. 13A, the player PL is located at a distance far from the sensor 162. The player PL can freely move in the moving area AR (play area) having the surface SA. In FIG. 13B, the player PL that is located at a distance far from the sensor 162 is closer to the sensor 162. Is approaching. That is, in FIG. 9, the player PL is moving in the forward direction and is approaching the sensor 162 and the display unit 190.

  FIG. 14A shows an example of a captured image of the sensor 162 (color sensor 164) when the player PL is located at a distance far from the sensor 162 as shown in FIG. In FIG. 14A, the entire player PL is in the imaging range.

  On the other hand, FIG. 14B is an example of a captured image of the sensor 162 when the player PL approaches the sensor 162 as shown in FIG. In FIG. 14B, when the player PL approaches, the player PL does not fit in the imaging range (sensing range) of the sensor 162, and the foot portion does not enter the captured image. 14B, even if skeleton information is acquired based on sensor information (captured image, depth information, etc.) from the sensor 162 in the state shown in FIG. Recognition is fraudulent. Therefore, when the character CHP is moved as shown in FIG. 9 based on the skeleton information, the character CHP moves incorrectly, and the generated image becomes an inappropriate and unnatural image. . For example, an image that looks as if the bone has been broken is generated, such as turning in the direction in which the foot of the character CHP is impossible.

  In order to prevent the occurrence of such a problem, in the present embodiment, when it is determined (predicted or inferred) that the skeleton recognition of the player PL (subject) based on the skeleton information will occur, the character CHP based on the skeleton information. The process of correcting the operation is performed. For example, the motion correction process for the character CHP is performed.

  Specifically, motion blending is performed as correction processing. For example, when it is determined that an illegality occurs in the skeleton recognition of the player PL, the motion blend of the skeleton motion based on the skeleton information and the reference motion of the character is performed.

  FIG. 15 is an explanatory diagram of motion blending. In FIG. 15, a skeleton motion MS is a motion based on skeleton information. That is, the motion of the character CHP defined by the skeleton recognition based on the skeleton information acquired from the sensor information. For example, the skeleton motion MS is a motion (posture) of the character CHP defined by a bone position (joint position), a bone rotation angle, and the like included in the skeleton information.

  On the other hand, the reference motion MF is prepared in advance as a motion (reference posture) serving as a reference for the character CHP. The data of the reference motion MF is stored in the motion data storage unit 174 in FIG. 1 and includes data such as the bone position (joint position) and the bone rotation angle in the reference posture. By applying this reference motion MF to the character CHP, the posture of the character CHP becomes a basic standing posture.

  As shown in FIG. 15, in this embodiment, when it is determined that an illegality occurs in the skeleton recognition of the player PL, the motion blend of the skeleton motion MS and the reference motion MF is performed, and the character CHP is converted by the obtained motion MT. Make it work. For example, by performing a motion blend of the bones BSR and BSL of the arms of the skeleton motion MS and the bones BFR and BFL of the arms of the reference motion, the positions and rotation angles of the bones BR and BL of the motion MT are obtained.

  If such motion blending is performed, even if there is an illegal bone in the skeleton motion MS, the reference motion MF composed of the appropriate bones is motion blended, so that the skeleton motion MS has an adverse effect due to the incorrect bone. Can be reduced.

  For example, in FIG. 12C, an irregularity occurs in the bone of the arm of the skeleton motion MS corresponding to the arm AMR of the player PL. In this regard, if the correction processing by the motion blend of this embodiment is performed, the appropriate arm bone (reference bone) of the reference motion MF is blended with the illegal arm bone of the skeleton motion MS. The adverse effects of the skeleton motion MS caused by unauthorized bones can be reduced. For example, even if the arm bone in the skeleton motion MS is bent in an inappropriate direction, the appropriate arm bone in the reference motion MF is blended, so that the arm bone is bent in an inappropriate direction. Can be appropriate. As a result, the situation in which the image of the character CHP becomes inappropriate or unnatural can be suppressed.

  Further, in FIG. 14B, the player PL's foot does not enter the imaging range, so that the skeleton motion MS corresponding to this foot has an injustice in the bone of the foot (leg). In this regard, if the correction process by the motion blend of this embodiment is performed, the leg bones of the reference motion MF are blended with the illegal leg bones of the skeleton motion MS, and the skeleton motion MS is blended. The adverse effects of incorrect bones can be reduced. For example, even if the bone of the foot in the skeleton motion MS is bent in an inappropriate direction, the bone of the appropriate foot in the reference motion MF is blended, so that the bone is bent in the incorrect direction. it can. As a result, the situation in which the image of the character CHP becomes inappropriate or unnatural can be suppressed.

  Further, according to the motion blending method, the influence of the skeleton motion MS based on the sensor information sensed by the player PL can be reflected in the blended motion MT. Accordingly, there is an effect that it is possible to realize a correction process that reflects the movement of the player PL. As will be described later, when motion blending is performed, it is desirable to change the blend rate of the skeleton motion MS and the blend rate of the reference motion MF in FIG. 15 as time elapses. In this way, the posture of the character CHP can be gradually changed from the posture based on the skeleton motion MS to the posture based on the reference motion MF as time elapses, and a motion image of the character that looks more natural can be generated. It becomes like this.

  In the present embodiment, correction processing is performed on bones that are determined to be fraudulent in player PL's skeleton recognition. Specifically, when performing the motion blend in FIG. 15, the motion blend is performed only for bones that are determined to be illegal in skeleton recognition. That is, motion blending is not performed on all bones that are the entire skeleton of the character CHP, but motion blending is performed only on illegal bones. For example, the blend process is performed only for the position (joint position) and the rotation angle of the bone that has become illegal. For example, in the case of FIG. 12C, motion blending is performed only on the bone corresponding to the arm AMR. In the case of FIG. 14B, motion blending is performed only on the bone corresponding to the foot. In this way, the processing load can be greatly reduced compared to the case where motion blending is performed on the entire skeleton.

  In the present embodiment, when it is determined that the skeleton recognition of the player PL (subject) is incorrect, the motion replacement that replaces the skeleton motion based on the skeleton information with the given motion may be performed as the correction processing. .

  For example, in FIG. 16, when it is determined that the skeleton recognition is illegal, the skeleton motion MS is replaced with the reference motion MF. Data of the reference motion MF, which is a motion replacement destination, is stored in the motion data storage unit 174 in FIG. When it is determined that the skeleton recognition is illegal, the data of the reference motion MF is read from the motion data storage unit 174 and the motion replacement process is performed.

  For example, in FIG. 16, the bones BSR2 and BSL2 of the leg of the skeleton motion MS are illegal. That is, as shown in FIG. 14B, when the foot portion of the player PL is not within the imaging range of the sensor 162, the foot cannot be detected by the sensor 162, and thus the skeleton recognition based on the skeleton information obtained from the sensor information. Injustice occurs in the foot bones. For example, when the skeleton information includes the reliability information of each bone, the reliability of the foot bone becomes a very low value. In other words, when the player PL's skeleton information is obtained and the player's PL foot cannot be detected as shown in FIG. 14B, the position and rotation angle of the foot bone are specified by the estimation process. In FIG. 14B, since the foot is not detected, it is notified by the reliability information included in the skeleton information that the reliability of the foot bone estimation process is very low. In this case, when the motion (posture) of the character CHP is determined using information on the bone of the foot with low reliability, the bones BSR2 and BSL2 of the foot are bent in an unnatural direction as in the skeleton motion MS of FIG. The resulting image becomes an inappropriate image.

  In this regard, in FIG. 16, motion replacement is performed from the skeleton motion MS to the reference motion MF. Thereby, the bones BSR2 and BLS2 in the bent direction are replaced with straight bones BFR2 and BFL2. That is, the appropriate bones BFR2 and BFL2 based on the reference motion MF are replaced. Therefore, it is possible to suppress the generation of an inappropriate image in which the foot is bent in an unnatural direction.

2.4 Specific Example of Correction Processing Next, a specific example of correction processing according to the present embodiment will be described. FIGS. 17A to 17D are diagrams illustrating specific examples of correction processing for preventing the occurrence of the problem described with reference to FIGS. 12A to 12C.

  First, as shown in FIG. 17A, the right shoulder position C3 and the left shoulder position C4 of the skeleton are acquired from the skeleton information described in FIG. 5, and a line LS connecting the right shoulder position C3 and the left shoulder position C4 is obtained. Then, based on the angle θ of the line LS, it is determined whether or not fraud occurs in the bone of one of the left arm and the right arm of the player PL.

  For example, in FIGS. 17B to 17D, the angle θ is an angle formed by the line LS and the visual line direction DV of the virtual camera VC. Here, the sensor 162 is assumed to be arranged and set at the position of the virtual camera VC in the object space.

  In FIG. 17B, as shown in FIG. 12A, since the player PL faces the sensor 162 in the front direction, the angle θ is 90 degrees. In this case, it is considered that both arms of the player PL are properly detected by the sensor 162 as shown in FIG. Therefore, it is determined that there is no fraud in the skeleton recognition based on the skeleton information.

  In FIG. 17C, the player PL rotates clockwise as shown in FIG. 12B, and the angle θ formed by the line LS and the line-of-sight direction DV is greater than 90 degrees. Also in this case, it is considered that both arms of the player PL are properly detected by the sensor 162, as shown in FIG. Therefore, it is determined that there is no fraud in skeleton recognition.

  On the other hand, in FIG. 17D, the player PL is further rotated with respect to FIG. 17C, and the angle θ is further increased. For example, θ ≧ 90 degrees + 60 degrees = 150 degrees. In this case, for example, as shown in FIG. 12C, one arm (for example, the right arm) of the player PL is hidden by, for example, the torso, etc. It is thought that it was not detected. Therefore, it is determined (predicted, guessed) that fraud has occurred in the skeleton recognition. Then, correction processing is performed on the bone of one arm (right arm). Specifically, as described with reference to FIG. 15, the motion blend of the skeleton motion MS based on the skeleton information and the reference motion MF is performed on the bone of one arm (right arm) determined to be illegal. More specifically, motion blending is performed only on the bones of one arm that are determined to be illegal in skeleton recognition.

  In this way, when the player PL rotates and one arm is hidden behind the player PL's body and the like, and it is predicted that the skeleton recognition will be fraudulent, Correction processing is performed. For example, motion blending with the bone (reference bone) of the reference motion MF is performed on the bone of the one arm. Accordingly, even when the player PL is rotated or the like and one arm is hidden behind the torso or the like, the character CHP that operates in an appropriate motion is displayed on the display unit 190 and generated. It is possible to suppress a situation in which the game image to be played becomes an inappropriate and unnatural image.

  In addition, when performing the motion blend of the skeleton motion MS and the reference motion MF, it is desirable to change the blend rate of the skeleton motion MS and the blend rate of the reference motion MF with time.

  For example, in FIG. 18, it is determined that the skeleton recognition based on the skeleton information is illegal at the timing t1. For example, as shown in FIG. 17D, the angle θ is equal to or greater than a predetermined angle (for example, 150 degrees or more), and it is predicted that an illegality occurs in the skeleton recognition. In this case, the motion blend of the skeleton motion MS (MS1, MS2, MS3, MS4...) And the reference motion MF (MF1, MF2, MF3, MF4...) Is performed, and the blend rate of the skeleton motion MS. αS and the blend rate αF of the reference motion MF are changed with the passage of time from the timing t1. For example, the blend rate αS of the skeleton motion MS is gradually decreased as time elapses from the timing t1. On the other hand, the blend rate αF of the reference motion MF is gradually increased as time elapses from the timing t1. This makes it possible to generate a character motion MT (MT1, MT2, MT3, MT4...) At each timing t1, t2, t3, t4.

  In this way, at the timing t1, the blend ratio of the skeleton motion MS is the largest, while the blend ratio of the reference motion MF is the smallest. As the time elapses from the timing t1, the blend ratio of the skeleton motion MS is gradually decreased, and the motion blend is performed so that the blend ratio of the reference motion MF is gradually increased. Therefore, the posture of the character CHP can be gradually changed from the skeleton motion MS based on the motion of the player PL to the reference motion MF based on the reference posture of the character CHP. Therefore, it is possible to realize a motion blend that can suppress the occurrence of problems caused by skeleton recognition fraud while reflecting the movement of the player PL to some extent.

  Further, in the present embodiment, when it is determined that an irregularity occurs in the foot bone of the skeleton information based on the distance of the player PL from the sensor 162, correction processing is performed on the foot bone. Specifically, as described with reference to FIG. 16, when it is determined that an injustice occurs in the bone of the foot, the bones BSR2 and BSL2 of the skeleton information are replaced with the reference bones BFR2 and BFL2.

  For example, in FIG. 19A, it is determined that the player PL is far from the sensor 162 (longer than a predetermined distance). In this case, the character CHP is operated using the skeleton motion MS based on the skeleton information obtained by detecting the movement of the player PL. Thereby, the movement of the character CHP according to the movement of the player PL can be realized.

  On the other hand, in FIG. 19B, the player PL is approaching the sensor 162, and it is determined that the distance of the player PL from the sensor 162 is short (shorter than the predetermined distance). In this case, motion replacement is performed to replace the skeleton motion MS based on the skeleton information with a given motion. Specifically, as shown in FIG. 16, the bones BSR2 and BSL2 of the skeleton motion MS are replaced with the bones BFR2 and BFL2 (reference bones) of the reference motion MF.

  In this way, when the player PL approaches the sensor 162 as shown in FIG. 19B, the player PL's foot does not enter the imaging range as shown in FIG. 14B, and the position of the foot is detected. In such a case, it is possible to suppress the occurrence of a problem such that the character CHP's feet turn in an unexpected direction. That is, by replacing the straight bones BFR2 and BFL2 (bones perpendicular to the ground) of the reference motion MF as shown in FIG. 16, the image of the character CHP becomes an inappropriate and unnatural image. It becomes possible to suppress.

2.5 Modifications Next, various modifications of the present embodiment will be described. For example, in FIGS. 15 and 16, when it is determined that the skeleton recognition based on the skeleton information is illegal, correction processing by motion blending or motion replacement is performed, but the present embodiment is not limited to this.

  For example, when it is determined that an illegality occurs in the player's skeleton recognition, a method of generating a game event prepared for the occurrence of an illegal skeleton recognition may be employed. FIG. 20 is a flowchart for explaining the processing of this method.

  First, the player's skeleton information is acquired (step S1). Then, it is determined whether or not fraud occurs in the skeleton recognition based on the skeleton information (step S2). For example, it is determined (predicted) whether fraud occurs in the skeleton recognition by the method described with reference to FIGS. 17 (A) to 17 (D), 19 (A), and 19 (B). When it is determined that fraud occurs, a game event for occurrence of fraud in skeleton recognition is generated (step S3). Information of this game event (game processing information) is stored in the storage unit 170 of FIG.

  For example, as shown in FIGS. 12 (A) to 12 (C), it is assumed that the player PL rotates his / her body, thereby causing skeleton recognition fraud. In this case, for example, when it is detected that the body of the player PL has rotated to some extent, a game event is generated that stops or returns the body of the player PL. For example, the guidance display that prompts the player PL to take a posture facing the display unit 190 without rotating the body is performed. For example, a character for guidance display is displayed on the display unit 190, or character information or icon information for guidance display is displayed on the display unit 190.

  In addition, as shown in FIGS. 13B and 14B, when the player PL approaches the sensor 162 and the parts such as the legs of the player PL cannot be detected, skeleton recognition fraud occurs. Suppose. In this case, a game event that stops the approaching movement of the player PL is generated. For example, a guidance display that prompts the player PL to stop without approaching or move backward is performed. For example, a character for guidance display is displayed on the display unit 190, or character information or icon information for guidance display is displayed on the display unit 190.

  FIGS. 21A to 21C are examples of motions in which the character CHP delivers a special technique. This special technique is, for example, a technique that is played out with a shout of “A”, “I”, “U”, “D”, “O”. FIG. 21A shows “A”, “I”, FIG. (B) corresponds to the timing of “U” and “D”, and FIG. 21 (C) corresponds to the timing of “O”.

  In this case, first, an instruction screen instructing the player PL to perform the movement of the special technique “Ieweo” is displayed on the display unit 190 of FIG. Then, from the timing of this instruction screen, the player PL performs a technical action corresponding to the motions of FIGS. 21 (A) to 21 (C). That is, from the front-facing posture as shown in FIG. 21A, the body is rotated as shown in FIG. 21B, and both hands are projected forward at the timing of the last “O”.

  The image generation system determines whether or not the player PL has started the above-described technique operation at the instruction timing on the instruction screen, and within a predetermined technique period (for example, 3 to 6 seconds), the player PL Determines whether or not the technique operation has been completed.

  Here, for example, when the player PL greatly rotates the body as shown in FIG. 21B, proper skeleton recognition cannot be performed as described in FIGS. 12C and 17D. End up. In this case, until the angle θ described in FIGS. 17A to 17D reaches a predetermined angle, the character CHP is a skeleton motion based on the skeleton information acquired by detecting the motion of the player PL. To work. When the angle θ exceeds the predetermined angle, the character CHP is operated by performing a motion correction process for switching from the skeleton motion to the special technique motion prepared for the special technique of “Ieweo”. As a result, motion destruction caused by skeleton recognition fraud can be suppressed. Then, if the player PL is performing a technique operation corresponding to FIGS. 21A to 21C within the technique period (3 to 6 seconds) described above, it is determined based on the skeleton information or the like. During this time, the motion correction process is continued.

  Then, if the player PL appropriately performs the technique action corresponding to FIG. 21C within the technique period, the character CHP makes a special attack (beam emission or the like), assuming that the special technique is established. Game effect processing is performed.

  In this case, as shown in FIG. 21C, the determination process of the action of sticking out both hands can be realized by using the depth information described with reference to FIG. For example, if depth information is detected at the timing of FIG. 21C so that it is determined that both hands have been pushed out, it is determined that the player PL has succeeded in the last “O” action of the special technique. 21A to 21C, the waist position of the player PL is also detected based on the skeleton information. Then, when it is detected that the player PL is performing a technique operation with a lower back, it is determined that the player PL is performing an appropriate technique operation.

  As described above, in FIGS. 21A to 21C, the correction process according to the game situation is performed as the correction process of the character motion. That is, FIGS. 21 (A) to 21 (C) are game situations in which the technique of the character CHP (the special technique of “Ieweo”) is selected, and therefore it is determined that fraud has occurred in the skeleton recognition. In the case of a motion, the motion is replaced with a correction motion corresponding to the selected technique. That is, in the situation where the upper body of the character CHP rotates and the arm cannot be seen as shown in FIG. 21B, the motion replacement from the skeleton motion based on the skeleton information to the correction motion corresponding to the technique is performed. Note that when the character CHP is performing an attack motion, the skeleton motion may be replaced with a prepared correction motion for attack. When the character CHP is performing a defensive motion, the skeleton motion may be replaced with a preparatory corrected motion for defensive.

  In the present embodiment, during the character motion (motion) correction process, the character determination process is performed based on the character motion after the correction process. For example, based on the motion after the correction process, various determination processes in the game progress are executed.

  For example, in the present embodiment, a commemorative photo shooting mode is prepared in which a player character and another character are taken together. An image photographed in this photographing mode is printed on a print medium by the printing unit 198 in FIG.

  In an image in such a shooting mode, for example, as shown by C1 in FIG. 22, if the player character CHP and the character CH overlap, it is not preferable as a commemorative photo composition, and the value of the commemorative photo is reduced. End up.

  Therefore, in the present embodiment, an overlap determination process is performed to determine whether or not the character CHP and the character CH overlap in C1 of FIG. If it is determined that the characters overlap, a rearrangement process is performed so that the character CHP and the character CH do not overlap, as indicated by C2 in FIG. Specifically, a rearrangement process for moving the position of the character CHP in the object space to the left side, for example, is performed so that the character CHP and the character CH do not overlap on the display screen. Note that it is possible to move both the character CHP and the character CH away from each other, or to move the character CH. As shown in FIG. 22, by detecting the overlap between characters and performing a rearrangement process so that they do not overlap, an image with an appropriate composition can be generated.

  When performing the overlap determination process of characters CHP and CH for such a rearrangement process, the overlap determination process is executed based on the action of the character CHP after the correction process.

  For example, the correction process period of FIG. 23A is a period during which the correction process of the motion of the character CHP as described with reference to FIGS. 15 and 16 is performed. In this correction process period, the determination process is performed with the motion after the correction process. For example, the overlap determination process described in FIG. 22 is performed using the motion after motion blending or the motion after motion replacement.

  For example, in FIG. 23B, the player PL rotates his / her body, so that the bone with the character CHP of the arm AML is in a state in which skeleton recognition cannot be performed (an illegal state). In this case, the correction process described with reference to FIGS. 15 and 16 is performed. For example, motion blending and motion replacement for the bones of the arm AML are performed. Then, based on the bone AML after the correction process, the overlap determination process described in FIG. 22 is performed. For example, it is determined whether or not the arm AML of the character CHP overlaps the character CH. In this way, since the overlap determination process can be realized by the action of the appropriate character after being corrected by the correction process, a more appropriate overlap determination process can be realized.

3. Detailed Processing Next, a detailed processing example of this embodiment will be described with reference to the flowcharts of FIGS.

  In FIG. 24, first, the player's skeleton information is acquired (step S1). Then, as described with reference to FIGS. 17A to 17D, it is determined whether or not the angle of the line LS connecting the right shoulder position C3 and the left shoulder position C4 has become an angle at which the arm bone becomes illegal. (Step S12). For example, when the angle θ as shown in FIG. 17D is reached, it is determined that the angle at which the bone of the arm is illegal is obtained. When the angle becomes illegal, the bone blending determined to be illegal is subjected to the motion blend of the skeleton motion MS and the reference motion MF until the motion blend period ends (steps S13 and S14). For example, the motion blending process described with reference to FIG. 15 is performed. At this time, as described with reference to FIG. 18, the process of changing the blend rate αS of the skeleton motion MS and the blend rate αF of the reference motion MF with the passage of time is performed until the motion blend period ends.

  In FIG. 25, first, the depth information of the player PL viewed from the sensor 162 is acquired, and the distance of the player PL from the sensor 162 is obtained (step S21).

  Next, as described with reference to FIGS. 19A and 19B, it is determined whether or not the distance of the player PL from the sensor 162 is a distance at which the bones of the foot become illegal (step S22). . That is, as shown in FIG. 14B, it is determined whether the distance is such that the foot of the player PL cannot be detected by the sensor 162. When such distance is reached, as described with reference to FIG. 16, motion replacement is performed to replace the bones BSR2 and BSL2 of the foot of the skeleton motion MS with the reference bones BFR2 and BFL2 of the reference motion MF (steps). S23).

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, in the specification or the drawings, terms (players, bones of arms and legs, skeleton information, etc.) described at least once together with different terms (subject, bones of a predetermined part, movement information, etc.) in a broader sense or synonymous The different terms can be used anywhere in the specification or drawings. In addition, the skeleton information acquisition process, the character motion process, the image generation process, the character motion correction process, the motion blend process, the motion replacement process, and the like are not limited to those described in this embodiment, and are equivalent to these. Such techniques are also included in the scope of the present invention.

PL player (subject), CHP character, IMF face image (part image),
MS skeleton motion, MF standard motion,
Motion after MT motion blending,
BSR, BSL, BFR, BFL, BR, BL Arm bone,
BSR2, BSL2, BFR2, BFL2 Foot bones,
C3 right shoulder position, C4 left shoulder position, LS line, θ angle,
100 processing units, 102 input processing units, 110 arithmetic processing units, 111 game processing units,
112 object space setting unit, 113 character processing unit,
114 subject information acquisition unit, 118 game score calculation unit,
119 Virtual camera control unit, 120 image generation unit, 130 sound generation unit,
132 print processing unit, 140 output processing unit, 150 payout slot, 152 coin insertion slot,
160 operation unit, 162 sensor, 164 color sensor, 166 depth sensor,
170 storage unit, 172 object information storage unit, 174 motion data storage unit,
178 Drawing buffer, 180 information storage medium, 190 display unit, 192 sound output unit,
194 I / F unit, 195 portable information storage medium, 196 communication unit, 198 printing unit

Claims (14)

A subject information acquisition unit that acquires skeleton information of a subject based on sensor information from the sensor;
A character processing unit that performs processing for moving a character based on the skeleton information of the subject;
An image generator for generating an image of the character;
Including
The character processing unit
An image generation system that performs correction processing of the action of the character based on the skeleton information when it is determined that an illegality occurs in the skeleton recognition of the subject based on the skeleton information. In claim 1,
The character processing unit
An image generation system characterized in that, when it is determined that an illegality occurs in the skeleton recognition of the subject, a motion blend of a skeleton motion based on the skeleton information and a reference motion of the character is performed as the correction processing. In claim 2,
The character processing unit
The image generation system characterized in that the motion blending is performed only on a bone that is determined to be illegal in the skeleton recognition of the subject. In claim 2 or 3,
The character processing unit
An image generation system, wherein the blend rate of the skeleton motion and the blend rate of the reference motion are changed over time. In claim 1,
The character processing unit
An image generation system characterized by performing, as the correction processing, motion replacement that replaces a skeleton motion based on the skeleton information with a given motion when it is determined that an illegality occurs in the skeleton recognition of the subject. In any one of Claims 1 thru | or 5,
The character processing unit
An image generation system, wherein the correction processing is performed on a bone that is determined to be fraudulent in the skeleton recognition of the subject. In any one of Claims 1 thru | or 6.
The character processing unit
The right shoulder position and the left shoulder position of the skeleton are obtained from the skeleton information of the subject, and based on the angle of the line connecting the right shoulder position and the left shoulder position, the bone of one of the left arm and the right arm of the subject is illegal. The image generation system according to claim 1, wherein the correction processing is performed on the bone of the one arm when it is determined that the image is generated. In claim 7,
The character processing unit
An image generation system characterized by performing a motion blend of a skeleton motion based on the skeleton information and a reference motion of the character for the bone of the one arm that is determined to be illegal. In any one of Claims 1 thru | or 8.
The character processing unit
An image generation system that performs the correction processing on the foot bone when it is determined that fraud occurs in the foot bone of the skeleton information based on the distance of the subject from the sensor. In claim 9,
The character processing unit
An image generating system, wherein, when it is determined that an injustice occurs in the foot bone, the foot bone of the skeleton information is replaced with a given reference bone. In any one of Claims 1 thru | or 10.
Including a game processing unit for performing game processing;
The game processing unit
An image generation system characterized by generating a game event prepared for occurrence of fraud in the skeleton recognition when it is determined that fraud occurs in the skeleton recognition of the subject. In any one of Claims 1 thru | or 11,
Including a game processing unit for performing game processing;
The character processing unit
An image generation system that performs the correction process according to a game situation as the correction process of the action of the character. In any one of Claims 1 to 12,
The character processing unit
During the period of the correction process, the image generation system is characterized in that a determination process for the character is performed based on the movement of the character after the correction process. A subject information acquisition unit that acquires skeleton information of a subject based on sensor information from the sensor;
A character processing unit that performs processing for moving a character based on the skeleton information of the subject;
As an image generation unit for generating an image of the character,
Make the computer work,
The character processing unit
A program for performing correction processing of the action of the character based on the skeleton information when it is determined that an illegality occurs in the skeleton recognition of the subject based on the skeleton information.