JP2011215968A - Program, information storage medium and object recognition system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a program for object recognition processing capable of efficiently and accurately performing processing for recognizing an object where a depth relation is considered, an information storage medium and an object recognition system.SOLUTION: An input image having a depth value of each pixel is acquired by irradiating the object with light and receiving the reflected light of the object, a specific area is set in the input image based on the depth value of each pixel of the input image, and the object recognition processing is performed in the specific area.

Description

  The present invention relates to a program, an information storage medium, and an object recognition system.

  2. Description of the Related Art Conventionally, there is an apparatus that performs a process of recognizing an object on an input image by obtaining a motion vector of each pixel of the input image and analyzing the input image (Patent Document 1).

  However, the conventional technique as shown in Patent Document 1 is made on the assumption that the moving object is a person, and in fact, the movement of an object (for example, “bird”) that is not a person is a subject. However, it was sometimes mistaken for human movement.

  In addition, there is a conventional technique for determining whether or not a moving object is a “person”, but such a conventional technique performs a recognition process using a recognition pattern of “person” in the entire input image. It was very inefficient and the processing load was high.

  Further, in the prior art, the object recognition processing is performed ignoring the depth relation of the object, so that there is a problem that the accuracy of the object recognition is lacking.

JP 2005-13750 A

  The present invention has been made in view of the above-described problems, and an object of the present invention is to perform processing for recognizing an object in consideration of depth relations efficiently and accurately. An object of the present invention is to provide an object recognition processing program, an information storage medium, and an object recognition system.

  (1) The present invention is a program for performing processing for recognizing an object, which obtains an input image having a depth value of each pixel by irradiating the object with light and receiving reflected light from the object, Based on the depth value of each pixel of the image, the computer functions as a region setting unit that sets a specific region in the input image and an object recognition processing unit that performs object recognition processing for recognizing an object. The present invention relates to a program for performing object recognition processing in a specific area. The present invention also relates to an information storage medium that is readable by a computer and stores (records) a program that causes the computer to function as each of the above-described units. The present invention also relates to an object recognition system including the above-described units.

  According to the present invention, since object recognition processing is performed in a concentrated manner in a specific area, an object can be recognized more accurately than in the past. Further, according to the present invention, since object recognition processing is performed in a specific area instead of the full screen, useless processing can be omitted and object recognition processing can be performed more efficiently than in the past. Furthermore, according to the present invention, a specific area is set in the input image based on the depth value of each pixel of the input image, and object recognition processing is performed in the specific area. Can recognize objects. For example, if an area that is regarded as being in a short distance in the input image is set as a specific area, an object existing in the short distance can be recognized more efficiently and accurately. In addition, according to the present invention, since the object recognition process is performed in the specific area, when another object is reflected in an area other than the specific area, it is possible to prevent a situation where the other object is erroneously recognized.

  (2) Further, in the program, the information storage medium, and the object recognition system of the present invention, the object recognition processing unit increases the accuracy of the object recognition process in the specific area more than the accuracy of the object recognition process in the area other than the specific area. The object recognition process may be performed in the specific area. According to the present invention, since the object recognition process is performed in the specific region with the accuracy of the object recognition process in the specific region being higher than the accuracy of the object recognition process in the region other than the specific region, the object can be recognized more accurately.

  (3) Further, in the program, the information storage medium, and the object recognition system of the present invention, the cycle in which the object recognition processing unit performs the object recognition process in the specific region is shorter than the cycle of the object recognition process in the region other than the specific region. Then, the object recognition process may be performed in the specific area. According to the present invention, since the cycle for performing the object recognition process in the specific region is shorter than the cycle of the object recognition process in the region other than the specific region, the object can be recognized more accurately.

  (4) Further, in the program, the information storage medium, and the object recognition system of the present invention, the object recognition processing unit increases the image accuracy of the specific area higher than the image accuracy of the area other than the specific area, and recognizes the object in the specific area. Processing may be performed. According to the present invention, the object recognition process is performed in the specific area with the image accuracy of the specific area higher than the image accuracy of the area other than the specific area, so that the object can be recognized more accurately.

  (5) In the program, the information storage medium, and the object recognition system of the present invention, the object recognition processing unit performs object recognition processing in an area other than the specific area, and performs object recognition processing in an area other than the specific area. You may make it make a period longer than the period which performs an object recognition process in a specific area | region. According to the present invention, it is possible to perform object recognition processing in a region other than the specific region. In addition, since the present invention makes the period for performing object recognition processing in areas other than the specific area longer than the period for performing object recognition processing in the specific area, machine power (the overall processing capacity of the computer) is mainly specified. It is possible to focus on the object recognition processing of the area, and it is possible to perform the object recognition processing of the specific area more accurately.

  (6) Further, in the program, the information storage medium, and the object recognition system of the present invention, the object recognition processing unit performs object recognition processing in an area other than the specific area, and the image accuracy of the area other than the specific area is determined in the specific area. The image accuracy may be lower than the image accuracy. According to the present invention, it is possible to perform object recognition processing in a region other than the specific region. In addition, since the image accuracy of the region other than the specific region is lower than the image accuracy of the specific region according to the present invention, the machine power can be mainly focused on the object recognition process of the specific region. Can be performed more accurately.

  (7) In the program, the information storage medium, and the object recognition system of the present invention, when the region setting unit sets a plurality of specific regions, the object recognition processing unit performs object detection for at least one specific region. Recognition processing may be performed. According to the present invention, when a plurality of specific areas are set, the object recognition process is performed for at least one specific area, so that an object can be recognized more accurately for at least one specific area.

  (8) In the program, the information storage medium, and the object recognition system of the present invention, the area setting unit gives priority to each specific area, and the object recognition processing section performs object recognition processing in each specific area. In addition, the period for performing the object recognition process in the specific area with low priority may be set longer than the period for performing the object recognition process in the specific area with high priority. According to the present invention, when recognizing an object in a plurality of specific areas, the period for performing the object recognition process in the specific area with low priority is set longer than the period for performing the object recognition process in the specific area with high priority. Therefore, the machine power can be focused on the specific area with higher priority, and the object recognition process can be performed efficiently.

  (9) In the program, the information storage medium, and the object recognition system of the present invention, the area setting unit gives priority to each specific area, and the object recognition processing section performs object recognition processing in each specific area. In addition, the image accuracy of the specific region with a low priority may be made lower than the image accuracy of the specific region with a high priority. According to the present invention, when recognizing an object in a plurality of specific areas, the period for performing the object recognition process in the low priority specific area, the image accuracy of the low priority specific area, and the high priority specific area Therefore, it is possible to focus the machine power on a specific area having a higher priority and perform object recognition processing efficiently.

  (10) In the program, the information storage medium, and the object recognition system of the present invention, the area setting unit assigns a priority to each specific area, and the object recognition processing unit performs an object in the low priority specific area. The object recognition process may be performed in a specific area with a high priority without performing the recognition process. According to the present invention, since the object recognition process is performed in the specific area with high priority without performing the object recognition process in the specific area with low priority, the object recognition process can be performed more efficiently.

  (11) In the program, the information storage medium, and the object recognition system of the present invention, the region setting unit sets a specific region in the input image based on the color information and the depth value of each pixel of the input image. It may be. According to the present invention, since the specific area is set in the input image based on the color information and the depth value of each pixel of the input image, it is possible to set the specific area where the object should be recognized more accurately.

  (12) Further, in the program, the information storage medium, and the object recognition system of the present invention, the region setting unit specifies an input image based on a depth value that is equal to or greater than a predetermined value among the depth values of each pixel of the input image. An area may be set. According to the present invention, since the specific area is set in the input image based on the depth value that is equal to or greater than a predetermined value among the depth values of each pixel of the input image, the specific area where the object should be recognized more accurately is set. be able to.

  (13) In the program, the information storage medium, and the object recognition system of the present invention, the object recognition processing unit may perform the object recognition processing in the specific area based on bone information. According to the present invention, since the object recognition process in the specific area is performed based on the bone information, the object can be recognized more accurately.

  (14) In the program, the information storage medium, and the object recognition system of the present invention, the object recognition processing unit may perform a process of recognizing an image change in the specific area at a predetermined timing. According to the present invention, a change in an image can be recognized with a simple process.

1 is an overview diagram of a first object recognition system of the present embodiment. The functional block diagram of the 1st object recognition system of this embodiment. Explanatory drawing of a motion vector. 4A to 4G are explanatory diagrams of motion vectors. The flowchart for calculating the direction of a motion vector. 6A, 6B, and 6C are explanatory diagrams of motion vectors. The figure for demonstrating the method of setting a specific area | region. Explanatory drawing for demonstrating the method of performing an object recognition process in a specific area | region. The figure for demonstrating the period of an object recognition process. 10A, 10B, and 10C are explanatory diagrams of motion vectors. The figure for demonstrating the method of setting a specific area | region. Explanatory drawing for demonstrating the method of performing an object recognition process in a specific area | region. The figure for demonstrating the period of an object recognition process. The figure for demonstrating the method of setting a specific area | region. Explanatory drawing regarding the priority of each specific area. Explanatory drawing for demonstrating the method of performing an object recognition process in a specific area | region. The figure for demonstrating the period of an object recognition process. The flowchart of 1st Embodiment. FIG. 6 is an overview diagram of a second object recognition system of the present embodiment. The functional block diagram of the 2nd object recognition system of this embodiment. 21A and 21B are explanatory diagrams of input images input to the input unit of the present embodiment. Explanatory drawing of the depth sensor of 2nd Embodiment. Explanatory drawing of the depth sensor of 2nd Embodiment. Explanatory drawing which shows the positional relationship of the position of the object in the real space of 2nd Embodiment, and an input part. 25A to 25D are explanatory diagrams of object recognition processing. The figure for demonstrating the method of setting a specific area | region. Explanatory drawing for demonstrating the method of performing an object recognition process in a specific area | region. 28A and 28B are explanatory diagrams for explaining a method of performing object recognition processing in a specific region. The figure for demonstrating the method of setting a specific area | region. Explanatory drawing for demonstrating the method of performing an object recognition process in a specific area | region. Explanatory drawing for demonstrating the method of performing an object recognition process in a specific area | region. The flowchart of 2nd Embodiment. 33A and 33B are explanatory diagrams of object motion recognition processing.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. 1. First embodiment 1-1. First Object Recognition System FIG. 1 is a schematic external view of a first object recognition system (a first game system and a first image generation system) in the first embodiment. The first object recognition system of the present embodiment includes a display unit 90 that displays a game image, an object recognition device 10 (game machine) that performs object recognition processing, game processing, and the like, and an input unit 20. As shown in FIG. 1, the input unit 20 is arranged around the display unit 90 (display screen 91) at a position associated with the display unit 90. For example, the input unit 20 may be disposed below the display unit 90 or may be disposed above the display unit 90.

  Then, the object recognition device 10 analyzes the input image acquired from the input unit 20 including the RGB camera (imaging unit) 21 in a stationary state, and recognizes an object on the input image. For example, as illustrated in FIG. 1, the object recognition device 10 compares the input image obtained by capturing an image of the player P with a recognition pattern of “person” (an example of an object), thereby determining “person” in the input image. Judge whether it can be recognized. Then, when the “person” is recognized, the object recognition apparatus 10 performs a process of recognizing the movement of the “person” and the gesture. Thereby, various game processes can be performed. In the first embodiment, a processing example of the first object recognition system using the input unit 20 will be described.

1-2. Configuration FIG. 2 is an example of a functional block diagram of the first object recognition system. In the first object recognition system, it is not necessary to include all the units in FIG. 2, and a configuration in which some of the units are omitted may be employed.

  The first object recognition system includes an object recognition device 10, an input unit 20, a display unit 90, and a speaker 92. The input unit 20 includes an RGB camera (imaging unit) 21, a processing unit 22, and a storage unit 23.

  The RGB camera (imaging unit) 21 forms an image of light emitted from an object on a light receiving plane of an image sensor using an optical system such as a lens, photoelectrically converts light and darkness of the image into an amount of electric charge, and sequentially reads out the light. Convert it into an electrical signal. Then, RGB (colorized) RGB image (an example of an input image) is output to the storage unit 23. The RGB camera 21 performs a process of outputting to the storage unit 23 at a predetermined cycle (for example, every 1/60 seconds). In addition, the processing unit 22 performs a process of transmitting an RGB image captured by the RGB camera 21 to the object recognition device 10. The storage unit 23 sequentially stores the RGB images output by the RGB camera 21.

  Next, the object recognition apparatus 10 of this embodiment is demonstrated. The object recognition apparatus 10 of this embodiment includes a storage unit 170, a processing unit 100, an information storage medium 180, and a communication unit 196.

  The storage unit 170 includes a main storage unit 171, a drawing buffer 172, a recognition pattern storage unit 173, an input image storage unit 174, and a difference image storage unit 175. The main storage unit 171 is a work area of the processing unit 100, and the drawing buffer 172 is a storage area for storing an image drawn by the image generation unit 120.

  The recognition pattern storage unit 173 is a storage area for storing a pattern and a template prepared in advance for specifying an object, and stores a pattern in association with each object. For example, visual features and pixel values themselves are stored in the recognition pattern storage unit 173 as recognition patterns.

  Note that the recognition pattern storage unit 173 may be a storage area constructed as a database. For example, one or more patterns may be stored in association with each object such as “person”, “hand”, “foot”, and “arm”.

  The input image storage unit 174 is a storage area for storing an input image acquired by the input unit 20 at a predetermined cycle in order to perform object recognition processing and motion vector calculation processing. The difference image storage unit 175 is a storage area for storing a difference pixel value obtained by taking a difference between pixel values of two images captured at different points in time in order to perform a motion vector calculation process.

  The processing unit 100 performs various processes according to the present embodiment based on data read from the program stored in the information storage medium 180. That is, the information recording medium 180 stores a program for causing a computer to function as each unit of the present embodiment (a program for causing a computer to execute processing of each unit).

  The communication unit 196 can communicate with other game machines via a network (Internet). The function can be realized by various processors, hardware such as a communication ASIC, a network interface card, or a program.

  Note that a program for causing a computer to function as each unit of the present embodiment may be distributed from a storage unit or an information storage medium included in a server to the information storage medium 180 (or the storage unit 170) via a network. Good. Use of such server information storage media is also within the scope of the present invention.

  The processing unit 100 (processor) performs object recognition processing, game processing, image generation processing, or sound control processing based on information acquired from the input unit 20, a program developed from the information storage medium 180 to the storage unit 170, or the like. I do.

  In particular, the processing unit 100 of the first object recognition system functions as an acquisition unit 110, a calculation unit 111, a region determination unit 112, an object recognition processing unit 113, a game calculation unit 114, an image generation unit 120, and a sound control unit 130. .

  The acquisition unit 110 performs processing for acquiring an input image (RGB image) captured by the RGB camera (imaging unit) 21.

  The calculation unit 111 calculates a motion vector of each pixel of the input image based on two input images captured at different times.

  The region setting unit 112 sets a specific region in the input image based on the motion vector of each pixel of the input image. The area setting unit 112 may set a plurality of specific areas. When a plurality of specific areas are set, priority (priority information) is assigned (set) to each specific area. That is, priority is given in specific area units. Note that the priority is information for performing object recognition processing of a specific area by giving priority to a specific area having a higher priority than a specific area having a lower priority than the specific area.

  The region setting unit 112 may set a specific region in the input image based on the motion vector and color information of each pixel of the input image. The region setting unit 112 may set a specific region in the input image based on a motion vector having a magnitude equal to or larger than a predetermined value among the motion vectors of the pixels of the input image.

  The object recognition processing unit 113 performs object recognition processing for recognizing an object. Here, the object recognition process for recognizing an object includes at least one of a process for recognizing the object itself, a process for recognizing the movement of the object, and a process for recognizing the gesture (shape or pose) of the object.

  For example, the object recognition processing unit 113 is stored in the recognition pattern storage unit 173 based on pixel information (at least one of pixel position coordinates, pixel color information, and pixel motion vector) of each pixel of the input image. It is determined whether or not “person” can be recognized using the recognition pattern of “person”.

  Specifically, a recognition pattern of the shape (shape, silhouette) of “person” is prepared, and it is determined whether or not the shape of the motion region indicated by the motion vector in the input image is the shape of “person”. When it is determined that the form is “person”, processing for determining that “person” is recognized is performed. On the other hand, when “person” cannot be recognized, it is determined whether or not the next object (“hand”) can be recognized using the recognition pattern of the next object (for example, “hand”). Then, the process of collating the input image with the next recognition pattern is performed until the object can be recognized.

  Further, when it is determined that “person” is recognized, the object recognition processing unit 113 performs processing for recognizing the movement of “person” based on pixel information of each pixel of the input image. Further, when it is determined that the “person” is recognized, the object recognition processing unit 113 performs a process of recognizing the “person” gesture based on the pixel information of each pixel of the input image.

  In particular, the object recognition processing unit 113 of the present embodiment performs object recognition processing in a specific area. For example, the object recognition processing unit 113 may perform the object recognition process in the specific area by increasing the accuracy of the object recognition process in the specific area higher than the accuracy of the object recognition process in the area other than the specific area.

  More specifically, the object recognition processing unit 113 performs the object recognition process in the specific region by setting the cycle of performing the object recognition process in the specific region to be shorter than the cycle of the object recognition process in the region other than the specific region. Further, the object recognition processing unit 113 performs object recognition processing in the specific area by increasing the image accuracy (definition) of the specific area higher than the image accuracy of the area other than the specific area.

  In addition, the object recognition processing unit 113 performs the object recognition process in an area other than the specific area, and makes the period for performing the object recognition process in the area other than the specific area longer than the period for performing the object recognition process in the specific area. In addition, the object recognition processing unit 113 performs object recognition processing in a region other than the specific region, and lowers the image accuracy of the region other than the specific region than the image accuracy of the specific region.

  In addition, when a plurality of (two or more) specific areas are set, the object recognition processing unit 113 performs object recognition processing for at least one specific area.

  In addition, when a plurality of specific areas are set, the object recognition processing unit 113 performs the object recognition process in each specific area and sets the period in which the object recognition process is performed in the specific area with a low priority. It may be made longer than the period for performing the object recognition process in a specific region having a high height.

  In addition, when a plurality of specific areas are set, the object recognition processing unit 113 performs object recognition processing in each specific area, and sets the image accuracy of the specific area with a low priority to the specific area with a high priority. The image accuracy may be lower than the image accuracy.

  In addition, when a plurality of specific areas are set, the object recognition processing unit 113 does not perform the object recognition process in the low priority specific area but performs the object recognition process in the high priority specific area. It may be.

  Further, the object recognition processing unit 113 may perform the object recognition processing in the specific region based on bone information.

  The game calculation unit 114 performs various game calculations. Here, the game calculation includes a process for starting a game when a game start condition is satisfied, a process for advancing the game, a process for placing an object such as a character or a map, a process for displaying an object, and a game result. There is a process or a process of ending a game when a game end condition is satisfied.

  For example, the game calculation unit 114 performs a game process based on input data from the input unit 20, a program, and the like. The game calculation unit 114 of the present embodiment performs a game calculation process based on the recognition result of the object recognition processing unit 113, for example. That is, when the object recognition processing unit 113 recognizes “person (player)”, the object recognition processing unit 113 recognizes the movement and gesture of the “person” and moves the “person” (the person moves to the left and right). The game calculation process may be performed based on an operation) or a gesture (specific pose).

  Note that the processing unit 100 may perform processing for arranging an object in the virtual space, processing for moving an object existing in the virtual space, and the like. For example, the processing unit 100 may perform processing for arranging an object in a virtual space (virtual three-dimensional space (object space), virtual two-dimensional space). For example, a display object such as a building, a stadium, a car, a tree, a pillar, a wall, and a map (terrain) is placed in the virtual space in addition to the character and the pointing object. Here, the virtual space is a virtual game space. For example, in the case of a virtual three-dimensional space, objects are arranged in three-dimensional coordinates (X, Y, Z) like a world coordinate system and a virtual camera coordinate system. Space.

  For example, the processing unit 100 arranges an object (an object composed of a primitive such as a polygon, a free-form surface, or a subdivision surface) in the world coordinate system. Also, for example, the position and rotation angle (synonymous with direction and direction) of the object in the world coordinate system is determined, and the rotation angle (X, Y, Z axis rotation) is determined at that position (X, Y, Z). Position the object at (Angle). Note that the processing unit 100 may perform a process of placing the scaled object in the virtual space.

  Further, the processing unit 100 may perform movement / motion calculation of an object in the virtual space. In other words, based on the input information received from the input unit, programs (movement / motion algorithm), various data (motion data), etc., the object is moved in the virtual space and the object is moved (motion, animation). Process. Specifically, object movement information (movement speed, movement acceleration, position, orientation, etc.) and movement information (position of each part constituting the object, or rotation angle) are obtained every frame (1/60 second). Processing to obtain sequentially is performed. Note that a frame is a unit of time for performing object movement / motion processing and image generation processing.

  The image generation unit 120 performs drawing processing based on the results of various processes performed by the processing unit 100, thereby generating an image and outputting it to the display unit 90. For example, the image generation unit 120 of the present embodiment generates an image that indicates the reference start timing and the reference determination period.

  The image generation unit 120 receives and inputs object data (model data) including vertex data (vertex position coordinates, texture coordinates, color data, normal vector, α value, etc.) of each vertex of the object (model). Based on the vertex data included in the object data, vertex processing (shading by a vertex shader) is performed. When performing the vertex processing, vertex generation processing (tessellation, curved surface division, polygon division) for re-dividing the polygon may be performed as necessary.

  In the vertex processing, according to the vertex processing program (vertex shader program, first shader program), vertex movement processing, coordinate transformation, for example, world coordinate transformation, visual field transformation (camera coordinate transformation), clipping processing, perspective transformation (projection transformation) ), Geometry processing such as viewport conversion is performed, and based on the processing result, the vertex data given to the vertex group constituting the object is changed (updated or adjusted).

  Then, rasterization (scan conversion) is performed based on the vertex data after the vertex processing, and the surface of the polygon (primitive) is associated with the pixel. Subsequent to rasterization, pixel processing (shading or fragment processing by a pixel shader) for drawing pixels (fragments forming a display screen) constituting an image is performed. In pixel processing, according to a pixel processing program (pixel shader program, second shader program), various processes such as texture reading (texture mapping), color data setting / change, translucent composition, anti-aliasing, etc. are performed, and an image is processed. The final drawing color of the constituent pixels is determined, and the drawing color of the perspective-transformed object is output (drawn) to the image buffer 172 (a buffer capable of storing image information in units of pixels; VRAM, rendering target). That is, in pixel processing, per-pixel processing for setting or changing image information (color, normal, luminance, α value, etc.) in units of pixels is performed. Thereby, an image that can be seen from the virtual camera (given viewpoint) in the object space is generated. Note that when there are a plurality of virtual cameras (viewpoints), an image can be generated so that an image seen from each virtual camera can be displayed as a divided image on one screen.

  The vertex processing and pixel processing are realized by hardware that enables polygon (primitive) drawing processing to be programmed by a shader program written in a shading language, so-called programmable shaders (vertex shaders and pixel shaders). Programmable shaders can be programmed with vertex-level processing and pixel-level processing, so that the degree of freedom of drawing processing is high, and expressive power is greatly improved compared to conventional hardware-based fixed drawing processing. Can do.

  The image generation unit 120 performs geometry processing, texture mapping, hidden surface removal processing, α blending, and the like when drawing an object.

  In the geometry processing, processing such as coordinate conversion, clipping processing, perspective projection conversion, or light source calculation is performed on the object. Then, the object data (positional coordinates of object vertices, texture coordinates, color data (luminance data), normal vector, α value, etc.) after geometry processing (after perspective projection conversion) is stored in the storage unit 170. .

  Texture mapping is a process for mapping a texture (texel value) stored in the storage unit 170 to an object. Specifically, the texture (surface properties such as color (RGB) and α value) is read from the storage unit 170 using texture coordinates or the like set (given) to the vertex of the object. Then, a texture that is a two-dimensional image is mapped to an object. In this case, processing for associating pixels with texels, bilinear interpolation or the like is performed as texel interpolation.

  As the hidden surface removal processing, hidden surface removal processing can be performed by a Z buffer method (depth comparison method, Z test) using a Z buffer (depth buffer) in which Z values (depth information) of drawing pixels are stored. . That is, when drawing pixels corresponding to the primitive of the object are drawn, the Z value stored in the Z buffer is referred to. Then, the Z value of the referenced Z buffer is compared with the Z value at the drawing pixel of the primitive, and the Z value at the drawing pixel is a Z value (for example, a small Z value) on the near side when viewed from the virtual camera. In some cases, the drawing process of the drawing pixel is performed and the Z value of the Z buffer is updated to a new Z value.

  α blending (α synthesis) is a translucent synthesis process (usually α blending, addition α blending, subtraction α blending, or the like) based on an α value (A value).

  For example, in α blending, the drawing color (overwriting color) C1 to be drawn in the image buffer 172 and the drawing color (background color) C2 already drawn in the image buffer 172 (rendering target) are set to α values. Based on this, a linear synthesis process is performed. That is, if the final drawing color is C, it can be obtained by C = C1 * α + C2 * (1−α).

  The α value is information that can be stored in association with each pixel (texel, dot), for example, plus alpha information other than color information. The α value can be used as mask information, translucency (equivalent to transparency and opacity), bump information, and the like.

  The sound control unit 130 performs sound processing based on the results of various processes performed by the processing unit 100, generates a game sound such as BGM, sound effect, or sound, and outputs the game sound to the speaker 92.

  Note that the terminal according to the present embodiment may be controlled so that the game can be played in a single player mode in which only one player can play or in a multiplayer mode in which a plurality of players can play. For example, in the case of controlling in the multiplayer mode, game processing may be performed by transmitting / receiving data to / from other terminals via a network, or one terminal may receive input information from a plurality of input units. Processing may be performed based on this.

  The information storage medium 180 (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (CD, DVD), magneto-optical disk (MO), magnetic disk, hard disk, and magnetic tape. Alternatively, it can be realized by hardware such as a memory (ROM).

  The display unit 90 outputs an image generated by the processing unit 100, and functions thereof are a CRT display, LCD (liquid crystal display), OELD (organic EL display), PDP (plasma display panel), touch panel display. Alternatively, it can be realized by hardware such as an HMD (head mounted display).

  The speaker 92 outputs sound to be reproduced by the sound control unit 130, and its function can be realized by hardware such as a speaker or headphones. Note that the speaker 92 may be a speaker provided in the display unit. For example, when a television (household television receiver) is used as the display unit, a television speaker can be used.

  In the present embodiment, the data stored in the recognition pattern storage unit 173, the input image storage unit 174, and the difference image storage unit 175 of the object recognition apparatus 10 are stored in the storage unit 23 of the input unit 20, The processing unit 22 of the input unit 20 may perform the processing of the calculation unit 111, the region setting unit 112, and the object recognition processing unit 113 of the embodiment.

1-3. Description of Motion Vector In this embodiment, an input image (RGB image) captured by the RGB camera 21 (imaging unit) is acquired. For example, as shown in FIG. 3, the digitized input images F1, F2, F3, and F4 are acquired from the RGB camera 21 at a predetermined period (for example, a period of 1/60 seconds) according to the drawing frame rate. Process.

  In the present embodiment, the same pixels are associated with each other in two input images captured at different times, and a motion vector (movement vector, movement direction) indicating the movement amount (movement amount) and the movement direction (movement direction) is obtained. Optical flow). A motion vector V in FIG. 3 indicates a vector directed to the pixel P2 of the input image F2 corresponding to the pixel P1 of the input image F1. In this embodiment, a motion vector is obtained for each pixel of the input image.

  In particular, in this embodiment, in order to reduce the processing load, a motion vector is obtained based on a difference image obtained by taking a difference between pixel values (luminance values and color values) from the input images F1 and F2. For example, as shown in FIG. 4A, an example in which a region having a pixel value “50” moves in the lower right direction will be described. In this embodiment, as shown in FIG. 4B, the previous image (input image F1) and the current image (input image F2) are stored in the input image storage unit 174, and the pixel values of the previous image and the current image are stored. And the difference image shown in FIG. 4C is stored in the difference image storage unit 175. Then, the amount of movement and the direction of movement of the pixel are obtained based on the difference image. That is, in the present embodiment, the absolute value of the difference pixel value of each pixel obtained by taking the difference between the previous image and the current image is set as the magnitude (motion amount) of the motion vector V of each pixel, and the previous image and the current image Based on the difference pixel value, an estimated orientation of each orientation (upward, downward, rightward, leftward) of the pixel is obtained.

The method for obtaining the estimated azimuth will be described in more detail with reference to the flowchart shown in FIG. First, in the present embodiment, two adjacent pixel values in a specific direction of the pixel P (x ₁ , y ₁ ) of the previous image are compared (step S1). For example, as shown in FIG. 4D, the pixel value of the pixel (x ₀ , y ₁ ) adjacent in the left-right direction of the pixel P (x ₁ , y ₁ ) of the previous image and (x ₂ , y ₁ ) Is compared with the pixel value of.

Then, it is determined whether or not the compared pixel values are equal (step S2). If the compared pixel values are not equal, the process proceeds to step S3. On the other hand, if the compared pixel values are equal, it is assumed that the pixel P (x ₁ , y ₁ ) is not moving, and the process is terminated. For example, since the pixel value of the pixel (x ₀ , y ₁ ) is different from the pixel value of the pixel (x ₂ , y ₁ ), the process proceeds to step S3.

Then, a pixel having a small pixel value is set to L, and a pixel having a large pixel value is set to G (step S3). For example, in the example of FIG. 4D, the pixel (x ₀ , y ₁ ) is L and the pixel (x ₂ , y ₁ ) is G.

Then, it is determined whether or not the difference pixel value of the difference image at the pixel P (x ₁ , y ₁ ) is 0 (step S4). If the difference pixel value is not 0, the process proceeds to step S5, where the difference When the difference pixel value of the image is 0, it is considered that the pixel P (x ₁ , y ₁ ) is not moving, and the process is terminated. For example, in the example of FIG. 4D, since the difference pixel value of the pixel P (x ₁ , y ₁ ) difference image is “−40”, the process proceeds to step S5.

Then, it is determined whether or not the difference pixel value is smaller than 0 (step S5). If the difference pixel value is smaller than 0, the process proceeds to step S6, and the process of selecting the P → G direction is performed (step S6). ). On the other hand, if the difference pixel value is not smaller than 0, the process proceeds to step S7, and a process of selecting the direction P → L is performed (step S7). In the example of FIG. 4D, the difference pixel value of the pixel P (x ₁ , y ₁ ) difference image is “−40”, which is smaller than 0. Therefore, the process proceeds to step S6 and the right direction of P → G is Selected as the estimated direction. The process ends here.

FIG. 4E is an example in which the estimated directions of the four directions (upward, downward, rightward, leftward) of the pixels are obtained for all the pixels. In the present embodiment, as shown in FIG. 4F, in each pixel, the sum of the estimated directions with respect to the four directions (upward, downward, rightward, leftward) of the pixel is determined as the pixel P (x ₁ , y The movement direction of ₁ ) may be used. Further, as shown in FIG. 4G, in each pixel, a direction obtained by smoothing the estimated direction including the eight surrounding pixels may be used as the motion direction.

  As described above, in this embodiment, the motion amount and motion vector of each pixel of the input image are obtained, but may be obtained by a so-called gradient method or block matching method. In the present embodiment, an RGB image is used as an input image, but a grayscale image having a luminance value may be used as an input image.

1-4. Object Recognition Processing In this embodiment, object recognition processing for recognizing an object on an image captured by an RGB camera is performed using a recognition pattern stored in advance in a storage unit. Here, the object recognition process for recognizing an object includes at least one of a process for recognizing the object itself, a process for recognizing the movement of the object, and a process for recognizing the gesture (shape or pose) of the object.

  For example, a recognition pattern of “person” stored in the recognition pattern storage unit 173 based on pixel information (at least one of pixel position coordinates, pixel color information, and pixel motion vector) of each pixel of the input image. Is used to determine whether or not “person” can be recognized.

  Specifically, an area of a pixel having a difference pixel value of 20 or more in the difference image or an area delimited by pixels having a difference pixel value greater than 0 in the difference image is set as a motion area, Using the (shape) recognition pattern, it is determined whether or not the shape of the motion region matches (matches) the shape of the “person” indicated by the recognition pattern. If it matches the shape of “person”, a process of determining that “person” has been recognized is performed. On the other hand, when “person” cannot be recognized, it is determined whether or not the next object can be recognized using the recognition pattern of the next object (for example, “hand”). Then, the process of collating the input image with the next recognition pattern is performed until the object can be recognized. As a result, when the object cannot be recognized in the motion region, the next motion region is specified and the object is recognized in the next motion region.

  In the present embodiment, (A) a difference image between two images may be taken every frame (1/60 second interval), and a motion region may be specified based on the difference pixel value of the difference image, or (B) A difference image between two images may be taken at 2 frames (1/30 second interval), and a motion region may be specified based on the difference pixel value of the difference image. (C) 10 frames (1/6 second interval) ), A difference image between the two images is taken, and the motion region may be specified based on the difference pixel value of the difference image.

  Note that the average area of the motion areas identified in (A), (B), and (C) may be identified as the motion area. For example, the average value A of the difference pixel values of the difference image in each frame during one second and the average value B of the difference pixel values of the difference image at 1/30 second intervals in the same one second are the same 1 You may make it set a motion area using the value which divided the average value C of the difference pixel value of the difference image in the interval of 1/6 second during 3 seconds by 3.

1-5. Processing for Recognizing Object in Specific Area In the present embodiment, a specific area is set, and object recognition processing is performed in the specific area. In this way, processing for recognizing an object efficiently and accurately can be performed. In addition, when another object is reflected in an area other than the specific area, it is possible to prevent a situation where the other object is erroneously recognized.

  First, in the present embodiment, a specific region is set in the input image based on the motion vector of each pixel of the input image. For example, based on the input image F1 as shown in FIG. 6A and the input image F2 shown in FIG. 6B (the input image F2 acquired 1/60 seconds after the input image F1), FIG. When the motion vector of each pixel between the input images F1 and F2 as shown in C) is obtained, the specific region is set based on the direction and magnitude of the motion vector between the input images F1 and F2. In other words, the specific area is set based on the difference pixel value of the difference image between the input images F1 and F2.

  For example, rule 1 is “a region where the average value of motion vector magnitude (difference pixel value) is 200 or more and the motion vector faces left or right in a given period (2 seconds)”. As shown in FIG. 7, the area determined based on the rule 1 is set as the specific area A1. For example, in the present embodiment, a rectangular area surrounding an area determined based on rule information such as rule 1 is set as the specific area A1. Note that the rules for setting the specific area are stored in the storage unit 70.

  In the present embodiment, the specific area may be set in the input image based on the motion vector and color information of each pixel of the input image. For example, “a pixel having a yellow color value, and an average value of a motion vector size (difference pixel value) is 200 or more in a given period (2 seconds), and the motion vector is left or The “area facing right” may be rule 1 ′, and the specific area A1 ′ may be set based on rule 1 ′.

  Note that once the specific area A1 is set, it is necessary to recognize an object in the specific area A1, and thus the specific area A1 is fixed for a given period (for example, 60 seconds) from the time when the specific area A1 is set. Then, the specific area A1 is updated (varied, reset) at a given period (for example, a period of 60 seconds).

  In this embodiment, the object recognition process is performed in the specific area A1 shown in FIG. That is, in the set specific area A1, the motion area S1 is set, and it is determined whether or not the shape of the motion area matches the recognition pattern. For example, in each pixel of the specific area A1, a set of pixels having a motion vector magnitude (difference pixel value) equal to or greater than a predetermined value is defined as a motion area S1.

  In this embodiment, for example, the shape of the motion area S1 of the specific area A1 is compared with the recognition pattern of “person” to determine whether the person is “person”. In the example of FIG. 7, it is determined that the pattern does not match the recognition pattern of “person”, and the shape of the motion area S1 of the specific area A1 is compared with the recognition pattern of “hand” to determine whether it is “hand”. . In the example of FIG. 7, it is determined that the recognition pattern matches the “hand” recognition pattern, and it is determined that “hand” has been recognized in the specific area A <b> 1.

  In the present embodiment, since machine power can be poured into a specific area A1 that is a part of the input image, the image accuracy of the specific area A1 may be increased to recognize an object. Image accuracy is the resolution of the image (total number of pixels in the image) and the quantization level of the image (range that pixels can take, gradation). The higher the resolution, the more precise the object can be recognized. . Further, the higher the quantization level, the wider the range of values that can be taken by the pixel value (difference pixel value), and the accuracy of setting the motion region more precisely can be increased.

  For example, as shown in FIG. 8, the resolution of the specific area A1 is increased, the motion vector of each pixel is recalculated in the specific area A1, and the motion area S1 ′ is calculated based on the motion vector of each pixel whose image accuracy is increased. May be set. In this way, for example, when “hand” is determined, the gesture (shape) of “hand” and the movement of “hand” can be recognized in more detail. In the example of FIG. 8, the degree of coincidence between the shape of the movement area S1 ′ of the specific area A1 and the three recognition patterns “goo”, “choki”, and “par” of “hand” is determined. It is determined that it most closely matches the recognition pattern.

  Further, in the present embodiment, as shown in FIG. 9, the cycle for performing the object recognition process in the specific area A1 may be shortened. For example, if a difference image between two input images acquired with a 1/6 second period is obtained before setting the specific area A1, it is acquired with a 1/60 second period after time t10 when the specific area A1 is set. A difference image between the two input images is obtained. That is, the difference image of the specific area A1 on the input image is obtained at 1/60 second intervals. In this way, the movement of the object can be recognized in more detail.

  As described above, in the present embodiment, machine power can be poured into the specific area A1, so that object recognition processing can be performed in detail for an object. In addition, in the present embodiment, there is an effect that the erroneous recognition of the object can be reduced by increasing the image accuracy for the specific region and further shortening the recognition cycle.

  For example, based on the difference image between the input image F10 shown in FIG. 10A and the input image F11 shown in FIG. 10B, a motion vector based on FIG. 10C is obtained. Suppose that the specific area B1 is set based on the rule 1 as shown in FIG. Here, when the image accuracy is not increased or when the recognition cycle is not shortened, it may be determined that the shape of the motion region S2 of the specific region B1 is not “bird” but “hand”.

  However, in the specific area B1, as shown in FIGS. 10A and 10B, since a bird actually flies, it is natural to determine that it is a “bird”. Therefore, in the present embodiment, as shown in FIG. 12, the image accuracy is increased for the specific region B1, the recognition cycle is shortened, and the object recognition process is performed correctly in detail. That is, the resolution of the specific area B1 is increased, the motion vector is recalculated, and the motion area S2 ′ is specified in the specific area B1. Then, it is determined that the shape of the motion region S2 ′ does not match the recognition pattern of “hand” but matches the recognition pattern of “bird”, and as a result, “bird” can be recognized in the specific region B1.

1-6. Relationship between the specific area and the area other than the specific area In the present embodiment, the object recognition process may be performed in an area other than the specific area A1 in FIG. When performing object recognition processing in a region other than the specific region, the image accuracy of the region other than the specific region is set lower than the image accuracy of the specific region. In this way, the recognition accuracy is inferior to that of the specific area A1, but when an object exists outside the specific area A1, the object can be recognized.

  In the present embodiment, the image accuracy of the region other than the specific region A1 may be made lower than the image accuracy of the specific region. Further, the cycle for performing the object recognition process in the region other than the specific region is set longer than the cycle for performing the object recognition process in the specific region A1. For example, as shown in FIG. 13, the object recognition process is performed at a period of 1/6 second in the area other than the specific area A1, and the object recognition process is performed at a period of 1/60 second in the specific area A1. In this way, the machine power (total processing capability of the computer) can be mainly focused on the object recognition process in the specific area A1, and the object recognition process in the specific area A1 can be performed more accurately.

1-7. Multiple specific areas In the present embodiment, multiple specific areas may be set. For example, rule 1 is “a region in which the average value of motion vectors (difference pixel values) is 200 or more and the motion vector faces left or right in a given period (2 seconds)”. When the rule 2 is “a region where the average value of motion vectors (difference pixel values) is 100 or more and the motion vector faces left or right in a given period (2 seconds)”, FIG. As shown in FIG. 14, the area determined based on the rule 1 is set as the specific area A1, and as shown in FIG. 14, the area determined based on the rule 2 is set as the specific area A2. In the present embodiment, three or more specific areas may be set.

  Then, object recognition processing is performed for at least one specific region. For example, the object recognition process may be performed for one of the specific areas A1 and A2 set based on the rule 1 and the rule 2, or the object recognition process may be performed for both the specific areas A1 and A2. May be. For example, when the object recognition process is performed in the specific area A2, the shape of the motion area S3 of the specific area A2 is compared with the recognition pattern of “person” to determine whether the person is “person”. In the example of FIG. 14, it is determined that the recognition pattern matches the “person” recognition pattern.

  In particular, in the present embodiment, when a plurality of specific areas are set, a priority is set (given) to each specific area. For example, as shown in FIG. 15, the priority is set in association with the ID of each rule (in association with each area). In this embodiment, the priority is determined according to the magnitude of the motion vector. For example, the average value of the motion vector magnitude (difference pixel value) of each pixel on the specific region may be calculated, and the priority may be set so that the higher the average value, the more dominant.

  In this embodiment, object recognition processing is performed based on the priority. For example, the image accuracy of a specific region with a low priority may be set lower than the image accuracy of a specific region with a high priority. That is, as shown in FIG. 16, the resolution of the specific area A2 is increased, and the resolution of the specific area A2 having a low priority is set lower than the resolution of the specific area A1 having a high priority. In this way, it is possible to recognize the object in detail by focusing mainly on the machine power for the specific area A1 with high priority, simplify the processing for the low priority area, and efficiently specify the specific area A1, The object recognition process of A2 can be performed.

  As shown in FIG. 16, when the resolution is increased for the specific area A2, the motion vector is recalculated in the specific area A2 with the increased resolution, and the shape of the motion area S3 ′ of the specific area A2 It is also possible to determine whether or not the person is a “person” by comparing with the recognition pattern “”. In the example of FIG. 16, it is determined that the recognition pattern matches the “person” recognition pattern.

  Similarly, for example, the period for performing the object recognition process in the specific area with low priority is set longer than the period for performing the object recognition process in the specific area with high priority. That is, as shown in FIG. 17, the period for performing the object recognition process in the specific area A2 with low priority is set longer than the period for performing the object recognition process in the specific area A1 with high priority. In this way, the object can be recognized in more detail with respect to the specific area A1 having a high priority.

  In the present embodiment, when a plurality of specific areas are set, the object recognition process is performed in the high priority specific area without performing the object recognition process in the low priority specific area. Also good. For example, the object recognition process may be performed in the specific area A1 having a high priority without performing the object recognition process in the specific area A2 having a low priority.

1-8. Flowchart Finally, the processing flow of this embodiment will be described with reference to FIG. First, based on the difference image of two input images, the motion vector of each pixel is calculated (step S10). Then, a specific area on the input image is set based on the motion vector (step S11). Then, the image accuracy of the specific area is increased, and the period for performing the object recognition process for the specific area is shortened (step S12). And the process which recognizes an object in a specific area is performed (step S13). The process ends here.

2. Second Embodiment Next, a second embodiment of the present embodiment will be described. The second embodiment is an application of the first embodiment. In the second embodiment, description of points that are common to the first embodiment will be omitted, and points that are different from the first embodiment and points that are added in the second embodiment will be described.

2-1. Second Object Recognition System FIG. 19 is a schematic external view of a second object recognition system (first game system, first image generation system) in the second embodiment. The second object recognition system of the present embodiment includes a display unit 90 that displays a game image, an object recognition device 50 (game machine) that performs object recognition processing, game processing, and the like, and an input unit 60. And as shown in FIG. 19, the input part 60 is arrange | positioned in the position linked | related with the display part 90 around the display part 90 (display screen 91). For example, the input unit 60 may be disposed below the display unit 90 or may be disposed above the display unit 90.

  The second object recognition system includes an input unit 60 (an example of a sensor) that can recognize the movement of the player P's hand or body. The input unit 60 includes a light emitting unit 610, a depth sensor 620, an RGB camera 630, and a sound input unit 640 (multi-array microphone). The input unit 60 is non-contact with the player P (object), and the player P's hand and body in real space. 3D position and shape information can be captured. In the second embodiment, a processing example of the second object recognition system using the input unit 60 will be described.

2-2. Configuration FIG. 20 is an example of a functional block diagram of the second object recognition system. In addition, description is abbreviate | omitted about the point which is common in the structural example of a 1st object recognition system, and the point which is different from the structural example of a 1st object recognition system is demonstrated. In the second object recognition system, it is not necessary to include all the units in FIG. 20, and a configuration in which some of the units are omitted may be employed.

  The second object recognition system includes an object recognition device 50, an input unit 60, a display unit 90, and a speaker 92.

  The input unit 60 includes a light emitting unit 610, a depth sensor 620, an RGB camera (imaging unit) 630, a sound input unit 640, a processing unit 650, and a storage unit 660.

  The light emitting unit 610 performs a process of irradiating an object (player, subject) with light. For example, the light emitting unit 610 irradiates a target object with light such as infrared rays from a light emitting element such as an LED.

  The depth sensor 620 includes a light receiving unit that receives reflected light from an object. The depth sensor 620 takes a difference between the amount of light received when the light emitting unit 610 emits light and the amount of light received when the light emitting unit 610 is not emitting light, thereby reflecting the object irradiated from the light emitting unit 610. Process to extract light. That is, as shown in FIG. 21A, the depth sensor 620 generates a reflected light image (an example of an input image) obtained by extracting reflected light of an object irradiated from the light emitting unit 610 in a predetermined unit time (for example, 1 (In units of 60 seconds), the process of outputting to the storage unit 660 is performed. The reflected light image can acquire the distance (depth value) from the input unit 60 to the object in units of pixels.

  The RGB camera (imaging unit) 630 forms an image of light emitted from an object (player P) on a light receiving plane of an image sensor by an optical system such as a lens, and photoelectrically converts light and darkness of the image light into an amount of charge. It is sequentially read out and converted into an electric signal. Then, RGB (colored) RGB image (an example of an input image) is output to the storage unit 660. For example, an RGB image as shown in FIG. 21B is generated. The RGB camera 630 performs a process of outputting to the storage unit 660 in a predetermined unit time (for example, in units of 1/60 seconds).

  The depth sensor 620 and the RGB camera 630 may receive light from a common light receiving unit. In this case, you may have two light-receiving parts. The light receiving unit for the depth sensor 620 and the light receiving unit for the RGB camera 630 may be different from each other.

  The sound input unit 640 performs voice recognition processing, and can be a multi-array microphone, for example.

  The processing unit 650 instructs the light emitting unit 610 to emit light, performs a process of transmitting the reflected light image output by the depth sensor 620 or the RGB image captured by the RGB camera 630 to the object recognition device 50, and the like. Do.

  The storage unit 660 sequentially stores the reflected light image output by the depth sensor 620 and the RGB image output by the RGB camera 630.

  Next, the object recognition apparatus 50 of this embodiment is demonstrated. The object recognition device 50 according to the present embodiment includes a storage unit 570, a processing unit 500, an information storage medium 580, and a communication unit 596.

  The recognition pattern storage unit 573 is a storage area for storing a pattern and a template prepared in advance for specifying an object, and stores a pattern in association with each object. For example, visual features and pixel values themselves are stored in the recognition pattern storage unit 573 as recognition patterns.

  Note that the recognition pattern storage unit 573 may be a storage area constructed as a database. For example, one or more recognition patterns may be stored in association with each object such as “person”, “hand”, “foot”, and “arm”.

  In particular, the recognition pattern storage unit 573 of the second object recognition system may store depth value information for specifying an object in the recognition pattern storage unit 573 as a recognition pattern.

  Further, the recognition pattern storage unit 573 of the second object recognition system may store a plurality of bone information (skeleton information, skeleton information). For example, bone information for recognizing a person, bone information such as a hand bone, an arm bone, and a foot bone may be stored in the recognition pattern storage unit 573 for each part constituting the person. . The bone information virtually defines the three-dimensional joint position of the human body and the rotation angle of the three-dimensional joint.

  The input image storage unit 574 is a storage area for storing an input image acquired by the input unit 20 at a predetermined cycle in order to perform object recognition processing and motion vector calculation processing. Further, the difference image storage unit 575 is a storage area for storing a difference pixel value obtained by taking a difference between pixel values of two images captured at different times in order to perform a motion vector calculation process.

  The processing unit 500 performs various processes according to the present embodiment based on data read from the program stored in the information storage medium 580. That is, the information recording medium 580 stores a program for causing a computer to function as each unit of the present embodiment (a program for causing a computer to execute processing of each unit).

  The communication unit 596 can communicate with other game machines via a network (Internet). The function can be realized by various processors, hardware such as a communication ASIC, a network interface card, or a program.

  Note that a program for causing a computer to function as each unit of the present embodiment may be distributed from a storage unit or an information storage medium included in the server to the information storage medium 580 (or the storage unit 570) via a network. Good. Use of such server information storage media is also within the scope of the present invention.

  The processing unit 500 (processor) performs game processing, image generation processing, or sound control processing based on information received from the input unit 60, a program developed from the information storage medium 580 to the storage unit 570, and the like.

  In particular, the processing unit 500 of the second object recognition system functions as an acquisition unit 510, a calculation unit 511, a region determination unit 512, an object recognition processing unit 513, a game calculation unit 514, an image generation unit 520, and a sound control unit 530. .

  The acquisition unit 510 performs processing for acquiring an input image such as an RGB image or a reflected light image from the input unit 50. That is, the acquisition unit 510 acquires the reflected light image (infrared reflection result) having the depth value of each pixel by receiving the reflected light of the object irradiated from the light emitting unit by the depth camera 620.

  The calculation unit 511 calculates a motion vector of each pixel of the input image based on two input images (RGB image and reflected light image) acquired at different times. For example, as with the calculation unit 111, the calculation unit 511 acquires an input image captured by the RGB camera 630, and based on two input images captured at different points in time, the motion vector of each pixel of the input image May be calculated.

  The region setting unit 512 sets a specific region in the input image based on the depth value of each pixel of the input image. Further, the area setting unit 512 may set a plurality of specific areas. When a plurality of specific areas are set, priority is given (set) to each specific area (priority is given in units of specific areas).

  The area setting unit 512 may set a specific area in the input image based on the color information and depth value of each pixel of the input image. The region setting unit 512 may set a specific region in the input image based on a depth value that is a predetermined value or more among the depth values of each pixel of the input image.

  The region setting unit 512 may set a specific region in the input image based on the motion vector and depth value of each pixel of the input image.

  The object recognition processing unit 513 performs object recognition processing for recognizing an object. Here, the object recognition process for recognizing an object includes at least one of a process for recognizing the object itself, a process for recognizing the movement of the object, and a process for recognizing the gesture (shape or pose) of the object.

  In particular, the object recognition processing unit 513 includes a recognition pattern storage unit based on pixel information (at least one of pixel depth value, pixel position coordinate, pixel color information, and pixel motion vector) of each pixel of the input image. It is determined whether or not “person” can be recognized using the recognition pattern of “person” stored in 573.

  Specifically, the object recognition processing unit 513 performs a process of cutting out a three-dimensional human silhouette (three-dimensional shape) based on pixel information of each pixel of the input image (for example, reflected light image, RGB image). . For example, based on the depth value (luminance value) of each pixel of the reflected light image and the color value of the RGB image, a process of cutting out a three-dimensional human silhouette is performed.

  And the object recognition process part 513 performs the said object recognition process in the said specific area | region based on bone information. For example, the object recognition processing unit 513 collates a plurality of bones (skeleton, skeleton) stored in the recognition pattern storage unit 573 with a silhouette, and sets a bone that most closely matches the silhouette. The object recognition processing unit 513 performs processing for calculating the motion of the set bone. That is, processing is performed in which the motion of the bone is regarded as the motion of the player P. In the present embodiment, processing for specifying the bone and acquiring the operation of the player P is performed for each frame. In this embodiment, when processing is performed in units of parts constituting a person, such as arm bones and leg bones, the extracted silhouette matches any of a plurality of bones in part units. Is determined, and the process of converting the movement of the part into the movement of the part of the player is performed.

  For example, the object recognition processing unit 513 prepares bone information of “people”, and a three-dimensional silhouette of a pixel group region having a luminance value (depth value) equal to or higher than a predetermined luminance value (predetermined depth value) from the reflected light image. , It is determined whether or not it matches the bone information of “person”. If the silhouette matches the bone information of “person”, a process of determining that “person” has been recognized is performed. On the other hand, when “person” cannot be recognized, it is determined whether or not the next object can be recognized using bone information of the next object (for example, “hand”). Until the object (“hand”) can be recognized, the input image and the next bone information are collated.

  In addition, the object recognition processing unit 513 performs processing for recognizing the motion of the silhouette based on bone information that matches the silhouette for each frame. For example, when it is determined that “person” is recognized based on the bone information of “person”, processing for recognizing the motion and gesture of “person” is performed based on the bone information.

  In addition, the object recognition processing unit 513 of the present embodiment performs object recognition processing in the specific area. For example, the object recognition processing unit 513 may perform the object recognition process in the specific area by increasing the accuracy of the object recognition process in the specific area higher than the accuracy of the object recognition process in the area other than the specific area.

  More specifically, the object recognition processing unit 513 performs the object recognition process in the specific region by setting the cycle of performing the object recognition process in the specific region to be shorter than the cycle of the object recognition process in the region other than the specific region. In addition, the object recognition processing unit 513 performs object recognition processing in the specific area by raising the image accuracy of the specific area higher than the image accuracy of the area other than the specific area.

  In addition, the object recognition processing unit 513 performs the object recognition process in a region other than the specific region, and makes the cycle of performing the object recognition process in the region other than the specific region longer than the cycle of performing the object recognition process in the specific region. The object recognition processing unit 513 performs object recognition processing in a region other than the specific region, and lowers the image accuracy of the region other than the specific region than the image accuracy of the specific region.

  Further, when a plurality of (two or more) specific areas are set, the object recognition processing unit 513 performs object recognition processing for at least one specific area.

  In addition, when a plurality of specific areas are set, the object recognition processing unit 513 performs the object recognition process in each specific area and sets the cycle in which the object recognition process is performed in the specific area with a low priority. It may be made longer than the period for performing the object recognition process in a specific region having a high height.

  In addition, when a plurality of specific areas are set, the object recognition processing unit 513 performs object recognition processing in each specific area, and sets the image accuracy of the specific area having a low priority to the specific area having a high priority. The image accuracy may be lower than the image accuracy.

  Further, when a plurality of specific areas are set, the object recognition processing unit 513 performs the object recognition process in the specific area with a high priority without performing the object recognition process in the specific area with a low priority. It may be.

  Further, the object recognition processing unit 513 may perform processing for recognizing a change in the image in the specific area at a predetermined timing.

  The game calculation unit 514 performs various game calculations. Here, the game calculation includes a process for starting a game when a game start condition is satisfied, a process for advancing the game, a process for placing an object such as a character or a map, a process for displaying an object, and a game result. There is a process or a process of ending a game when a game end condition is satisfied.

  For example, the game calculation unit 514 performs a game process based on input data from the input unit 60, a program, and the like. The game calculation unit 514 of the present embodiment performs a game calculation process based on the recognition result of the object recognition processing unit 513, for example. That is, when the object recognition processing unit 513 recognizes “person (player)”, the object recognition processing unit 513 recognizes the movement and gesture of “person” and moves the “person” (the person moves to the left and right). The game calculation process may be performed based on an operation) or a gesture (specific pose).

  Note that, like the processing unit 100, the processing unit 500 may perform processing for arranging an object in the virtual space, processing for moving an object existing in the virtual space, and the like.

  Further, like the processing unit 100, the processing unit 500 may perform movement / motion calculation of an object in the virtual space.

  Similar to the image generation unit 120, the image generation unit 520 performs drawing processing based on the results of various processes performed by the processing unit 500, thereby generating an image and outputting the image to the display unit 90.

  Similar to the sound control unit 130, the sound control unit 530 performs sound processing based on the results of various processes performed by the processing unit 500, generates game sounds such as BGM, sound effects, and sounds, and the speaker. 92.

  Note that the object recognition system of the present embodiment may be controlled so that the game can be played in a single player mode in which only one player can play or in a multiplayer mode in which a plurality of players can play. For example, in the case of controlling in the multiplayer mode, game processing may be performed by transmitting / receiving data to / from other terminals via a network, or one terminal may receive input information from a plurality of input units. Processing may be performed based on this.

  An information storage medium 580 (a computer-readable medium) stores programs, data, and the like, and functions as an optical disk (CD, DVD), a magneto-optical disk (MO), a magnetic disk, a hard disk, and a magnetic tape. Alternatively, it can be realized by hardware such as a memory (ROM).

  Note that the data stored in the recognition pattern storage unit 573, the input image storage unit 574, and the difference image storage unit 575 of the object recognition device 50 are stored in the storage unit 660 of the input unit 60, and the calculation unit of the present embodiment. The processing unit 650 of the input unit 60 may perform the processing of the area setting unit 512 and the object recognition processing unit 513.

2-3. Description of Input Unit The input unit 60 of the second object recognition system includes a depth sensor 620 and an RGB camera 630, does not require an input device such as a controller, and performs image processing on an object (player, player's hand, etc.). By doing so, input can be accepted. As a result, various game processes that are not possible in the past can be performed. First, the depth sensor 620 and the RGB camera 630 of the input unit 60 will be described.

2-3-1. Depth Sensor A depth sensor 620 of this embodiment will be described with reference to FIG. First, as shown in FIG. 21, the light emitting unit 610 included in the input unit 60 emits light whose intensity varies with time in accordance with the timing signal. The emitted light is applied to a player P (an example of an object) located in front of the light source.

  The depth sensor 620 receives the reflected light of the light emitted from the light emitting unit 610. That is, the depth sensor 620 generates a reflected image image obtained by extracting the spatial intensity distribution of the reflected light. For example, the depth sensor 620 takes the difference between the amount of light received when the light emitting unit 610 emits light and the amount of light received when the light emitting unit 610 is not emitting light, thereby obtaining an object of light from the light emitting unit 610. A reflected light image is obtained by taking out the reflected light by. The value of each pixel of the reflected light image corresponds to the distance (depth value) from the position GP of the input unit 60 of the depth sensor 620 to the object. Note that the position GP of the input unit 60 is synonymous with the position of the depth sensor 620 and the light receiving position of the depth sensor 60.

  For example, in the example of FIG. 22, since the hand portion of the player P is closest to the position GP of the input unit 60, the region indicating the hand of the player P as shown in FIG. A reflected image that is a portion (high luminance portion) is obtained.

  In the present embodiment, for each pixel of the reflected light image, a pixel having a luminance value (light reception amount, pixel value) that is equal to or greater than a predetermined value is extracted as a pixel close to the position GP of the input unit 60. For example, when the gradation of the reflected light image is 256 gradations, pixels having a predetermined value (for example, 200) or more are extracted as high-luminance portions.

  The reflected light image obtained by this depth sensor is related to the distance (depth value) to the object. That is, as shown in FIG. 23, when the player P is located 1 meter away from the position GP of the input unit 60, the reflected light image of the reflected light image is larger than when the player P is located 2 meters away from the position GP. The region of the hand has high brightness (the amount of received light increases). Further, when the player P is located 2 meters away from the position GP of the input unit 60, the hand region portion of the reflected light image has a higher luminance than when the player P is located 3 meters away from the position GP. (The amount of received light increases).

  Based on such a principle, in the present embodiment, the position of the player P in the real space is calculated based on the luminance value of the pixel extracted as a high luminance part in the reflected light image. For example, the pixel having the highest luminance value in the reflected light image is used as a feature point, and the distance from the position GP to the player P is calculated based on the luminance value of the feature point. Note that the feature point may be a barycentric pixel of a hand region specified based on a shape pattern or a motion vector prepared in advance. Note that when the high-luminance portion is wide in the reflected light image, it can be determined that the object is present near the input unit, for example, compared to when the high-luminance portion is narrow.

  In the present embodiment, the position of the object in the real space with the input unit 60 as a reference can be specified based on the reflected light image. For example, when there is a feature point at the center of the reflected light image, it can be specified that the object is located in the emission direction of the light source of the input unit 60. When the feature point is at the upper part of the reflected light image, it can be specified that the object is at the upper part with reference to the input unit 60. Further, when the feature point is at the lower part of the reflected light image, it can be specified that the object is at the lower part with reference to the input unit 60. When the feature point is on the left side of the reflected light image, it can be determined that the object is on the right side with respect to the input unit 60 (as viewed from the front of the input unit (on the light source side)). When the feature point is on the left side of the reflected light image, it can be specified that the object is on the right side (as viewed from the front (light source side) of the input unit) with reference to the input unit 60. Thus, in this embodiment, the positional relationship between the object and the input unit 60 can be specified based on the reflected light image.

  In the present embodiment, the moving direction of the object in real space can be specified based on the reflected light image. For example, when there is a feature point at the center of the reflected light image and the brightness value of the feature point is high, it can be specified that the object is approaching the light source direction of the input unit 60. When the feature point is moving from the upper part to the lower part of the reflected light image, it can be determined that the object is moving from the upper part to the lower part with reference to the input unit 60. Further, when the feature point is moving from the left part to the right part of the reflected light image, it can be specified that the object is moving from the right part to the left part with reference to the input unit 60. Thus, in this embodiment, the moving direction of the object can be specified based on the input light 60 based on the reflected light image.

  Note that the reflected light of the object greatly decreases as the distance of the object from the position GP of the input unit 60 increases. For example, the amount of received light per pixel of the reflected light image decreases in inverse proportion to the square of the distance to the object. Therefore, when the player P is located about 20 meters away from the input unit 60, the amount of light received is so small that the reflected light from the player P can be ignored, and a high-luminance portion that can identify the player P is extracted. Can not do it. In such a case, it may be controlled that there is no input. In addition, when a high luminance part cannot be extracted, a warning sound may be output from a speaker.

2-3-2. Description of RGB Camera In the present embodiment, an RGB image is acquired as input information by an RGB camera (imaging unit) 630. Since the RGB image corresponds to the reflected light image, it is possible to improve the accuracy of the object motion vector (movement vector), the specific region setting process, and the object recognition process.

  The RGB camera 630 of the second embodiment can perform the same processing as the RGB camera 21. That is, also in the second embodiment, as shown in FIG. 3, two input images captured at different times are associated with the same pixel, and the amount of movement (movement amount) and the direction of movement (movement direction) ) Indicating a motion vector (movement vector, optical flow). Also in the second embodiment, a motion vector may be obtained based on a difference image obtained by taking a difference between pixel values (luminance values, color values) from the input images F1 and F2.

  In the present embodiment, the depth sensor 620 can specify the distance (depth value) from the input unit 60 to the object, and the position coordinates of the feature points of the high-luminance portion on the two-dimensional plane of the reflected light image or the RGB image (X, Y), the motion vector of the feature point can be extracted. Therefore, based on the distance (Z) from the input unit 60 to the object and the position coordinates (X, Y) of the reflected light image and the RGB image, the position Q of the object in the real space is specified. it can. Therefore, in the present embodiment, as shown in FIG. 24, the specific region setting process, the object, and the object are calculated based on the distance L from the position GP of the input unit 60 to the object position Q calculated based on the reflected light image. Recognition processing and game calculation can be performed.

2-4. Object Recognition Process In this embodiment, an object recognition process for recognizing an object on an input image (reflected light image, RGB image) is performed using a recognition pattern stored in advance in a storage unit. Here, the object recognition process for recognizing an object includes at least one of a process for recognizing the object itself, a process for recognizing the movement of the object, and a process for recognizing the gesture (shape or pose) of the object.

  The object recognition processing of the second object recognition system will be described in detail. For example, as shown in FIG. 25A, a reflected light image (see FIG. 25A) in which reflected light of an object irradiated from a light emitting unit is received by a depth camera 620. Infrared reflection result) is acquired.

  Then, as shown in FIG. 25B, a process of cutting out a silhouette (shape) is performed based on the luminance value of each pixel of the reflected light image. That is, a process of extracting (extracting) an area having a pixel value that is equal to or greater than the predetermined brightness value (200) among the brightness values of each pixel of the reflected light image as the silhouette ST1.

  Then, each of the plurality of bone information (skeleton, skeleton) stored in the recognition pattern storage unit 573 is compared with the silhouette ST1, and the bone information that most closely matches the silhouette ST1 is set. For example, as shown in FIG. 25C, bone information BO1, BO2, and BO3 related to “person” is stored in the recognition pattern storage unit 573, and it is determined that the silhouette ST1 most closely matches the bone information BO1. , A process of determining that “person” has been recognized is performed.

  On the other hand, when “person” cannot be recognized, it is determined whether or not the next object (“hand”) can be recognized using bone information of the next object (for example, “hand”). Then, the process of collating the input image with the next bone information is performed until the object can be recognized. In the example of FIG. 25D, it is determined that the silhouette ST1 most closely matches the bone information BO1 of “person” among the bone information BO1, BO2, and BO3, and it is determined that “person” has been recognized.

  In this embodiment, processing for recognizing the motion of an object and processing for recognizing the gesture of the object are performed based on bone information matching the silhouette for each frame. For example, at time s0, it is determined that “person” is recognized based on the bone information BO1 of “person”, and at time s10, ten frames after time s0, based on the bone information BO2 of “person”. If it is determined that “person” is recognized, and it is determined that “person” is recognized based on the bone information BO3 of “person” at time s20, 20 frames after time s1, each bone information BO1, BO2 By using BO3 as a key frame and interpolating the movement between the key frames, the movement of the “person” from the time point s1 to the time point s20 can be recognized. For example, when “person” is recognized based on the bone information BO3 of “person” at time s20 and at time s30 10 frames after s20, between s20 and s30 It is recognized that the object (person) is performing a gesture based on the bone information BO3.

  In this embodiment, since the object is recognized by determining whether the three-dimensional bone information matches the silhouette extracted in the three-dimensional manner, the three-dimensional object, the movement of the object, and the gesture are determined. Can be recognized.

  In the present embodiment, the object recognition process can also be performed on a part basis (hand, arm, face, foot) constituting a person. In such a case, a plurality of bones may be stored in advance in the recognition pattern storage unit 573 for each part, and it may be determined which of the plurality of bones the extracted silhouette matches.

  In addition, the second object recognition system sets a motion region based on the difference image, like the first object recognition system, and the silhouette of the motion region is recognized by using the recognition pattern of the shape of “person”. It may be determined whether or not it matches (matches) the shape of the “person” indicated by.

2-5. Processing for Recognizing Object in Specific Area In the present embodiment, a specific area is set, and object recognition processing is performed in the specific area. In this way, processing for recognizing an object efficiently and accurately can be performed. In addition, when another object is reflected in an area other than the specific area, it is possible to prevent a situation where the other object is erroneously recognized.

  That is, in this embodiment, a specific area is set in the input image based on the depth value of each pixel of the reflected light image (input image) shown in FIG. For example, when a reflected light image as shown in FIG. 21A is obtained, a high-luminance portion is set as a specific region based on the depth value.

  For example, the rule 10 is “a region where the average value of the luminance values of the pixels of the reflected light image is not less than a threshold value (eg, not less than 220) in a given period (2 seconds)”, as shown in FIG. The area determined based on the rule 10 is set as the specific area C1. Note that a rule for setting the specific area is stored in the storage unit 570.

  In the present embodiment, the specific area may be set in the input image based on the depth value and color information of each pixel of the input image. For example, a rule is defined as “a pixel having a yellow color value, and the average value of the luminance values of the reflected light image is not less than a threshold value (eg, not less than 220) in a given period (2 seconds)”. The specific area C1 ′ may be set based on the rule 10 ′.

  Note that once the specific area C1 is set, it is necessary to recognize an object in the specific area C1, and thus the specific area C1 is fixed for a given period (for example, 60 seconds) from the time when the specific area C1 is set. Then, the specific area C1 is updated (changed or reset) at a given period (for example, a period of 60 seconds).

  In the present embodiment, the object recognition process is performed in the specific area C1 shown in FIG. That is, in the set specific area C1, the silhouette ST2 of the pixel area where the luminance value of the pixel of the reflected light image is equal to or higher than a threshold value (for example, 220 or higher) is cut out, and which bone information is matched? Can be judged. It should be noted that the threshold for cutting out the silhouette in the specific area may be the same as the threshold of the rule for setting the specific area.

  For example, the silhouette ST2 of the specific area C1 is compared with the bone information of “person”, and it is determined whether or not the silhouette ST2 and the bone information of “person” match. In the example of FIG. 26, it is determined that the silhouette ST2 does not match the bone information of “person”. Then, the silhouette ST2 of the specific area C1 is compared with the next bone information, for example, “hand” bone information, and it is determined whether or not the silhouette ST2 and the “hand” bone information match. In the example of FIG. 26, it is determined that the silhouette ST2 matches the bone information of “hand”, and it is determined that “hand” has been recognized in the specific region C1.

  In the present embodiment, machine power can be applied to a specific area C1 that is a part of the input image. Therefore, an object may be recognized by increasing the image accuracy of the specific area C1. Image accuracy refers to image resolution (total number of pixels in the image), image quantization level (range that pixels can take, gradation), and number of bone information joints. The higher the resolution, the more detailed the object Can be recognized. Further, the higher the quantization level, the wider the range of values that can be taken by the pixel value (difference pixel value), and the silhouette can be extracted in more detail.

  For example, as shown in FIG. 27, the resolution of the specific area C1 is increased, the luminance value (depth value) of each pixel is recalculated in the specific area C1, and the silhouette is calculated based on the luminance value of each pixel of the specific area C1. Cut out ST2 '. For example, a pixel region having a luminance value of 220 or more is cut out as a silhouette ST2 ′. In this way, for example, when “hand” is determined, the gesture (shape) of “hand” and the movement of “hand” can be recognized in more detail.

  Further, the number of joints in the bone information of the specific area C1 may be increased to recognize the object in more detail. More specifically, the object may be recognized by comparing the silhouette ST2 ′ extracted in FIG. 27 with bone information with an increased number of joints. For example, as shown in FIG. 28B, when it is determined that the silhouette ST2 ′ matches the bone information BOH1 of “hand”, it is determined that “hand” is recognized in the specific region C1.

  In the present embodiment, a plurality of bone information groups with different numbers of joints are prepared in advance in association with objects. For example, low, medium, and high levels are provided, and bone information groups are prepared so that the number of joints increases as the level increases. For example, in the case of “person”, the bone information group having 13 joints is set to the low level, the bone information group having 28 joints is set to the medium level, and the bone information group having 56 joints is set to the high level. And In the case of “hand”, the bone information group with one joint is set to a low level, the bone information group with five joints is set to a medium level, and the bone information group with 15 joints is set to a high level. It is said.

  In this embodiment, when the object recognition process is performed in the specific area, the object recognition process is performed using at least the bone information group of the middle level or higher. For example, when the object recognition process is performed in the specific area C1, the silhouette of the specific area C1 is compared with each bone information of the bone information group of the medium-level “person” to determine whether or not the person is “person”. to decide.

  It is determined that the silhouette ST2 ′ of the specific area C1 shown in the example of FIG. 27 does not match the “person” bone information. Next, the silhouette ST2 ′ of the specific area C1 and the high-level “hand” bone information group. Each bone information is compared, and it is determined that the silhouette ST2 ′ of the specific area C1 matches the bone information BOH1, and it is recognized that the object in the specific area C1 is a “hand”.

  In the present embodiment, when the object itself is recognized in the specific area C1, the gesture or movement of the object in the specific area C1 is determined based on the bone information group of the number of joints that recognized the object itself in the specific area C1. You may make it perform the process which recognizes. For example, when the “hand” itself is recognized in the specific area C1 based on the bone information of the high-level “hand”, the “hand” of the specific area C1 is based on the bone information group of the high-level “hand”. The process of recognizing the gesture or movement of “” may be performed.

  In the present embodiment, as shown in FIG. 9, the cycle for performing the object recognition process in the specific area C1 may be shortened. For example, if the reflected light image is acquired at a 1/6 second period before setting the specific area C1, it may be acquired at a 1/60 second period after t10 when the specific area C1 is set. . In this way, the movement of the object can be recognized in more detail.

  As described above, in the present embodiment, machine power can be poured into the specific area C1, and therefore object recognition processing can be performed in detail on an object. In addition, in the present embodiment, there is an effect that the erroneous recognition of the object can be reduced by increasing the image accuracy for the specific region and further shortening the recognition cycle.

2-6. Relationship between the specific region and the region other than the specific region In the present embodiment, the object recognition process may be performed in a region other than the specific region C1 in FIG. When performing object recognition processing in a region other than the specific region, the image accuracy of the region other than the specific region is set lower than the image accuracy of the specific region. In this way, the recognition accuracy is inferior to that of the specific area C1, but the object can be recognized when an object exists outside the specific area C1.

  In the present embodiment, the image accuracy of the region other than the specific region C1 may be lower than the image accuracy of the specific region. In addition, the period for performing the object recognition process in an area other than the specific area may be longer than the period for performing the object recognition process in the specific area. For example, as shown in FIG. 13, the object recognition process is performed at a period of 1/6 second in the area other than the specific area C1, and the object recognition process is performed at a period of 1/60 second in the specific area C1. In this way, the machine power (the overall processing capability of the computer) can be mainly focused on the object recognition process in the specific area C1, and the object recognition process in the specific area C1 can be performed more accurately.

2-7. Multiple specific areas In the present embodiment, multiple specific areas may be set. For example, the rule 10 is “a region where the average value of the luminance values of the reflected light image is not less than a threshold value (eg, not less than 220) in a given period (2 seconds)”, and “a given period (2 seconds)”. , The area where the average value of the luminance value of the reflected light image is equal to or greater than a threshold value (for example, 200 or more) is set as the rule 20, the area determined based on the rule 10 as shown in FIG. While setting as the specific area C1, as shown in FIG. 29, the area determined based on the rule 20 is set as the specific area C2. In the present embodiment, three or more specific areas may be set. That is, in the present embodiment, since the luminance value of the reflected light image is related to the depth (distance to the object), the specific area is set according to the distance relationship of the object.

  Then, object recognition processing is performed for at least one specific region. For example, the object recognition process may be performed on one of the specific areas C1 and C2 set based on the rule 10 and the rule 20, or the object recognition process may be performed on both the specific areas C1 and C2. May be.

  For example, as shown in FIG. 29, when the object recognition process is performed for the specific area C2, a pixel area having a luminance value of 200 or more in the specific area C2 is cut out as a silhouette ST3. Then, the object is specified by comparing the silhouette ST3 with the bone information. As shown in FIG. 31, for example, when the silhouette ST3 and the “person” bone match, it is determined that “person” has been recognized in the specific region C2.

  In the present embodiment, the priority is set for each specific area in the same manner as in the first object recognition system. In the present embodiment, the priority is determined based on the depth value. For example, the average value of the depth value of each pixel on the specific region may be calculated, and the priority may be set so that the higher the average value, the more dominant. In this way, the object recognition process can be preferentially performed on an object closer to the input unit 60.

  In this embodiment, object recognition processing is performed based on the priority. For example, the image accuracy of a specific region with a low priority may be set lower than the image accuracy of a specific region with a high priority.

  For example, if the priority of the specific area C1 is 1 and the priority of the specific area C2 is 2, as shown in FIG. 30, the resolution of the specific area C2 is increased and the specific area C2 having a low priority is selected. The resolution is set lower than the resolution at which the object recognition process is performed in the specific area C1 having a high priority.

  For the specific area C1 with high priority, as shown in FIG. 28B, the number of joints is increased from a low level to a high level, and object recognition processing is performed based on the high-level bone information group. For the specific area C2 that is lower than the specific area C1, as shown in FIG. 31, the number of joints is increased from the low level to the medium level, and the object recognition process is performed based on the medium level bone information group.

  For example, as shown in FIG. 29, when performing the object recognition process for the specific area C2, the resolution of the specific area C2 is increased, and the luminance value (depth value) of each pixel is recalculated in the specific area C2. Then, a pixel region having a luminance value of 200 or more is cut out as a silhouette ST3 ′. Then, the object is specified by comparing the silhouette ST3 'with the bone information. As shown in FIG. 31, for example, when the silhouette ST3 ′ and the “people” bone BO5 match, “people” can be recognized in the specific region C2.

  By doing this, it is possible to recognize the object in detail by pouring machine power more than the specific area C1 with high priority, simplify processing for the low priority, and efficiently specify the specific areas C1 and C2. Object recognition processing can be performed.

  Similarly, for example, the period for performing the object recognition process in the specific area with low priority is set longer than the period for performing the object recognition process in the specific area with high priority. That is, as shown in FIG. 17, the period for performing the object recognition process in the specific area C2 with low priority is set longer than the period for performing the object recognition process in the specific area C1 with high priority. In this way, the object can be recognized in more detail with respect to the specific area C1 having a high priority.

  In the present embodiment, when a plurality of specific areas are set, the object recognition process is performed in the high priority specific area without performing the object recognition process in the low priority specific area. Also good. That is, the object recognition process may be performed in the specific area C1 having a high priority without performing the object recognition process in the specific area C2 having a low priority.

2-8. Flowchart Finally, the processing flow of this embodiment will be described with reference to FIG. First, a specific area on the input image is set based on the depth value of each pixel of the input image (step S20). And while improving the image precision of a specific area, the period which performs the object recognition process of a specific area is shortened (step S21). And the process which recognizes an object in a specific area is performed (step S22). The process ends here.

3. Game Calculation Processing Example In the present embodiment, various games can be performed by performing the object recognition processing described above.

3-1. Example of baseball game 3-1-1. Example of Performing Game Calculation of Baseball Game with First Object Recognition System In this embodiment, an example of performing game calculation of a baseball game with the first object recognition system will be described. First, in this embodiment, a player character is placed in the object space. Then, the player's movement and gesture are recognized, and based on the recognized player's movement and gesture, an action process is performed in which the player character throws the ball as a pitcher.

  For example, at the timing when the player character stands on the mound (predetermined area) in the object space, the average value of the magnitudes of motion vectors (difference pixel values) in rule 2 (a given period (2 seconds)) in the input image The motion region S3 indicated by the motion vector is extracted from the specific region A2 that is determined based on the motion vector of 100 or more and the motion vector facing left or right. Then, it is determined whether or not the shape of the motion region S3 matches the “person” pattern (“person” shape, “person” color information). When human recognition processing is performed, in the specific area A2, a motion area S3 ′ is set based on the motion vector of each pixel with increased image accuracy, and the shape of the motion area S3 ′ and “hand” The degree of coincidence of the recognition pattern is determined.

  In the present embodiment, when “person” is recognized in the specific area A2, it is next determined whether or not “hand” can be recognized based on the input image. For example, in the input image, rule 1 (region in which the average value of the motion vector (difference pixel value) is 200 or more and the motion vector faces left or right in a given period (2 seconds)) It is determined whether or not the shape of the motion area S1 of the specific area A1 determined based on the pattern matches the “hand” pattern (“hand” shape, “hand” color information). For example, when hand recognition processing is performed, a motion region S1 ′ is set based on the motion vector of each pixel whose image accuracy is increased in the specific region A1, and the shape of the motion region S1 ′ and “hand” The degree of coincidence of the recognition pattern is determined.

  When “hand” is recognized in the specific area A1, next, the gesture of “hand” is recognized. In the present embodiment, a throwing form on which the player character throws the ball is determined based on the recognized “hand” gesture. For example, the degree of coincidence between the shape of the movement area S1 ′ of the specific area A1 and the three recognition patterns “goo”, “choki”, and “par” of “hand” is determined. If the shape of the motion area S1 ′ matches the recognition pattern of “Goo”, the player character's throwing form is determined to be “straight”, and the shape of the motion area S1 ′ is a recognition pattern of “Choki”. Is determined to be “fork”, and when the shape of the movement area S1 ′ matches the recognition pattern of “par”, the player character's pitch form is determined to be “fork”. Decide on “Slider”.

  Then, the movement of the “hand” is recognized based on the direction of the motion vector of each pixel in the specific area A1 (or the movement area S1 ′), and the motion calculation of the player character in the object space is performed based on the movement of the “hand”. Do. For example, in the specific area A1 in which “hand” is recognized, it is detected that the direction of the motion vector is the downward direction on the screen and the magnitude of the motion vector is greater than or equal to a predetermined value (eg, the difference pixel value is greater than or equal to 250). In this case, the player character existing in the object space performs an action of throwing the ball with the determined throwing form.

  As described above, when the base object game calculation is performed by the first object recognition system, the object recognition, gesture, and movement of “person” and “hand” are recognized in a specific area instead of the full screen. Therefore, useless processing can be omitted and processing can be performed efficiently.

3-1-2. Example of Performing Game Calculation of Baseball Game with Second Object Recognition System Next, an example of performing game calculation of a baseball game with the second object recognition system of the present embodiment will be described. First, in this embodiment, a player character is placed in the object space, the player's movement and gesture are recognized, and the player character performs a motion process of throwing a ball as a pitcher based on the recognized player movement and gesture.

  For example, at the timing when the player character stands on the mound in the object space, the average value of the luminance value of the reflected light image is not less than a threshold value (for example, not less than 200 in the given period (2 seconds)) in the input image. ) Is determined as the specific region C2, and it is determined whether or not the silhouette ST3 of the specific region C2 matches the bone information of “person”. For example, when “person” is recognized, the silhouette ST3 ′ is set based on each pixel information by increasing the image accuracy of the specific area C2, and the degree of coincidence between the shape of the silhouette ST3 ′ and the bone information of “person” Judging.

  When “person” is recognized in the specific area C2, it is next determined whether or not “hand” can be recognized based on the input image. For example, in the input image, an area determined based on rule 10 (an area where the average value of the luminance value of the reflected light image is not less than a threshold value (for example, not less than 220) in a given period (2 seconds)) is specified. It is set as the region C1, and it is determined whether or not the shape of the silhouette ST2 of the specific region C1 matches the bone information of “hand”. For example, the silhouette ST2 ′ is set based on each pixel information by increasing the image accuracy of the specific area C1, and the degree of coincidence between the shape of the silhouette ST2 ′ and the bone information of “hand” is determined.

  Then, when “hand” is recognized in the specific area C1, next, the gesture of “hand” is recognized. For example, in the present embodiment, a throwing form on which the player character throws the ball is determined based on the recognized gesture. For example, the degree of coincidence between the shape of the silhouette ST2 ′ of the specific area C1 and the three pieces of bone information “goo”, “choki”, and “par” of “hand” is determined. When the shape of the silhouette ST2 ′ matches the bone information of “Goo”, the pitching form of the player character is determined to be “straight”, and the shape of the silhouette ST2 ′ matches the bone information of “Cho” In this case, the pitch form of the player character is determined as “fork”, and when the shape of the silhouette ST2 ′ matches the bone information of “par”, the pitch form of the player character is determined as “slider”.

  In the present embodiment, the movement of the “person” is recognized based on the bone information that matches the silhouette ST3 ′. For example, in the present embodiment, the bone information of “person” matching the silhouette ST3 ′ updated at a predetermined cycle is used as a key frame, and the motion between the key frames is interpolated to recognize and recognize the operation of “person”. Based on the motion, motion processing of the player character existing in the object space is performed. For example, when it is recognized that the movement of throwing a “person” ball is recognized in the specific area C2 in which “person” is recognized, the player character existing in the object space moves to the ball based on the determined throwing form. Do the action of throwing.

  As described above, when the base object game calculation is performed by the second object recognition system, the object recognition, gesture, and movement of “person” or “hand” are recognized in a specific area instead of the full screen. Therefore, useless processing can be omitted and processing can be performed efficiently.

3-2. Example of competitive game 3-2-1. Example of performing game calculation of a battle game using the first object recognition system An example of performing game calculation of a battle game using the first object recognition system will be described. First, in this embodiment, a player character is placed in the object space. Then, the player's movement and gesture are recognized, and based on the recognized player's movement and gesture, the player character determines an attack technique, a defense technique, and the like for the enemy character.

  For example, when a battle game between a player character and an enemy character is started in the object space, the shape of the motion area S3 indicated by the motion vector is recognized in the specific area A2 determined based on the rule 2 in the input image. It is determined whether or not the pattern matches the “person” pattern (“person” shape, “person” color information) stored in the storage unit 173. For example, the motion region S3 ′ is set based on the motion vector of each pixel whose image accuracy is improved in the specific region A2, and the degree of coincidence between the shape of the motion region S3 ′ and the recognition pattern of “person” is determined.

  When “person” is recognized in the specific area A2, it is next determined whether or not “hand” can be recognized based on the input image. For example, whether the specific area A1 is set based on the rule 1 in the input image and the shape of the motion area S1 of the specific area A1 matches the “hand” pattern (“hand” shape, “hand” color information). Judge whether or not. For example, the motion region S1 ′ is set based on the motion vector of each pixel whose image accuracy is improved in the specific region A1, and the degree of coincidence between the shape of the motion region S1 ′ and the recognition pattern of “hand” is determined.

  When “hand” is recognized in the specific area A1, next, the gesture of “hand” is recognized. For example, in the present embodiment, the technique of the player character is determined based on the recognized gesture. For example, the degree of coincidence between the shape of the movement area S1 ′ of the specific area A1 and the three recognition patterns “goo”, “choki”, and “par” of “hand” is determined. For example, when the shape of the motion area S1 ′ matches the recognition pattern of “Goo”, the technique of the player character is determined to be “punch”, and the shape of the motion area S1 ′ matches the recognition pattern of “choki” In this case, the technique of the player character is determined to be “tackle”, and if it matches the recognition pattern of “par”, the technique of the player character is determined to be “throw technique”.

  Then, the movement of the “hand” is recognized based on the direction of the motion vector of each pixel in the specific area A1 (or the movement area S1 ′). For example, in the specific area A1 in which “hand” is recognized, it is detected that the direction of the motion vector is the downward direction on the screen and the magnitude of the motion vector is greater than or equal to a predetermined value (for example, the difference pixel value is greater than or equal to 250). In this case, based on the determined “skill”, the player character existing in the object space performs an action to attack the enemy character.

  In the present embodiment, processing for moving the player character may be performed based on the recognized movement of the player's person or hand (motion calculation may be performed). In this way, for example, the punching action performed by the player can be reflected in the player character. For example, when the movement of the “hand” is recognized based on the direction of the motion vector of each pixel in the specific area A1 (or the movement area S1 ′), the movement is present in the object space based on the movement of the “hand”. The player character's hand (arm) motion processing is performed. For example, in the specific area A1 in which “hand” is recognized, when it is detected that the direction of the motion vector is the downward direction on the screen and the magnitude of the motion vector is a predetermined value or more, the “hand” of the player is It is determined that the player character has moved from top to bottom, and motion processing is performed in which the player character's hand (arm) is swung down from top to bottom.

  As described above, when the game calculation of the battle game is performed by the first object recognition system, the object recognition, gesture, and movement of “person” or “hand” are recognized in a specific area instead of the full screen. Therefore, useless processing can be omitted and processing can be performed efficiently.

3-2-2. Example of Performing Competitive Game Game Calculation in Second Object Recognition System Next, an example of performing a competitive game game calculation in the second object recognition system of the present embodiment will be described. First, in this embodiment, the player's movement and gesture are recognized, and the player character determines an attack technique, a defense technique, and the like for the enemy character based on the recognized player movement and gesture.

  For example, when a battle game between a player character and an enemy character is started in the object space, the silhouette ST3 cut out in the specific area C2 determined based on the rule 20 in the input image matches the bone information of “person”. Judge whether to do. For example, when “person” is recognized, the silhouette ST3 ′ is set based on each pixel information by increasing the image accuracy of the specific area C2, and the degree of coincidence between the shape of the silhouette ST3 ′ and the bone information of “person” Judging.

  If the silhouette ST3 matches the bone information of “person” and recognizes that it is “person”, it is next determined whether or not “hand” can be recognized based on the input image. For example, the specific region C1 is set based on the rule 10 in the input image, and it is determined whether or not the shape of the silhouette ST2 of the specific region C1 matches the bone information of “hand”. For example, the silhouette ST2 ′ is set based on each pixel information by increasing the image accuracy of the specific area C1, and the degree of coincidence between the shape of the silhouette ST2 ′ and the bone information of “hand” is determined.

  Then, when “hand” is recognized in the specific area C1, next, the gesture of “hand” is recognized. For example, in the present embodiment, the technique of the player character is determined based on the recognized gesture. For example, when the silhouette ST2 ′ matches the bone information of “Goo”, the skill of the player character is determined as “punch”, and when the silhouette ST2 ′ matches the bone information of “Cho”, the technique of the player character is determined. Is determined to be “tackle”, and if the silhouette ST2 ′ matches the bone information of “par”, the technique of the player character is determined to be “throw technique”.

  Then, the movement of the “hand” is recognized based on the bone information of the specific area C1 (or silhouette ST2 ′). For example, in the specific region C1 in which “hand” is recognized, it is detected that the bone of “hand” is protruding forward with respect to the depth sensor (moving the hand forward from the back). The player character existing in the object space attacks the enemy character based on the determined “skill”.

  In the present embodiment, the player's movement or gesture may be recognized, and a process for moving the player character may be performed based on the player's movement or gesture (motion calculation is performed).

  For example, when the movement of the “person” is recognized based on the bone information of the specific area C2 (or silhouette ST3 ′), the motion processing of the player character existing in the object space is performed based on the movement of the “person”. Do. For example, in the present embodiment, the bone information of “person” matching the silhouette ST3 ′ updated at a predetermined cycle is used as a key frame, and the motion between the key frames is interpolated to recognize and recognize the operation of “person”. Based on the motion, motion processing of the player character existing in the object space is performed. In this way, for example, the punching action performed by the player can be reflected in the player character.

  As described above, when the game calculation of the battle game is performed by the second object recognition system, the object recognition, gesture, and movement of “person” or “hand” are recognized in a specific area instead of the full screen. Therefore, useless processing can be omitted and processing can be performed efficiently.

3-3. Other Games This embodiment can be applied to various games such as action games, racing games, shooting games, sports games, competitive games, role playing games, simulation games, music performance games, and dance games.

  For example, when applied in each game, the object recognition process may be performed in a specific area at a predetermined timing according to the progress of the game. For example, the object recognition process is performed in the specific area at the timing when the player object to be operated by the player encounters the enemy object in the object space (the timing at which the position of the player object and the position of the enemy object are in a predetermined positional relationship). It may be.

  Further, in a racing game, a shooting game, or the like, it may be recognized that the player is operating a handle, a control stick or the like. For example, when a specific area for recognizing a hand (or an arm or the like) is set in a racing game or the like and the hand can be recognized in the specific area, the movement of the hand or the gesture of the hand is detected in the specific area. Perform recognition processing.

  In a music game or a dance game, the player's action may be recognized to determine whether or not the player's action matches a predetermined action. For example, when a specific area for recognizing a person is set and the person can be recognized in the specific area, a process of recognizing the movement of the person and the gesture of the person is performed in the specific area.

  In the present embodiment, various systems (terminals) such as an arcade game system, a home game system, a large attraction system in which a large number of players participate, a simulator, a multimedia terminal, a system board for generating game images, and a mobile phone ).

4). Application Example In this embodiment, the processing of the processing unit 100 of the first object recognition system may be applied to the processing unit 500 of the second object recognition system, or the processing of the processing unit 500 of the second object recognition system. The processing may be applied to the processing unit 100 of the first object recognition system. For example, in the first object recognition system, object recognition processing may be performed using bone information. Specifically, bone information related to “person” is stored in the recognition pattern storage unit 173 of the first object recognition system. Then, the plurality of bones (skeleton, skeleton) stored in the recognition pattern storage unit 173 are collated with the motion region (two-dimensional silhouette) set based on the pixel information of each pixel of the input image. Set bones that match the region.

  And in this embodiment, the process which calculates the motion of the set bone is performed. That is, processing is performed in which the motion of the bone is regarded as the motion of the player P. In the present embodiment, a process of specifying the bone and acquiring the action of the player P is performed at predetermined intervals (for example, every frame).

  In the present embodiment, the object recognition process can also be performed on a part basis (hand, arm, face, foot) constituting a person. In such a case, a plurality of bones may be stored in advance in the recognition pattern storage unit 173 for each part, and it may be determined which of the plurality of bones the extracted motion region matches.

5. Description of Processing for Recognizing Object Movement In this embodiment, processing for recognizing a change in an image may be performed at a predetermined timing. That is, in the present embodiment, processing for recognizing the movement of an object may be performed based on input images acquired at two different times. For example, the movement of an object (movement of an object) may be recognized simply by taking a difference between two input images. In this way, it is possible to recognize the movement and movement of the object with a simple process.

  More specifically, as shown in FIGS. 33A and 33B, the input image F20 acquired at the timing T20 and the time T21 (for example, 1 from the time T20) at a predetermined timing after the time T20 has elapsed. The motion may be recognized by taking a difference from the input image F21 acquired at time T21 after 2 seconds). That is, a difference value between the total value of each pixel value of the input image F20 and the total value of each pixel value of the input image F21 is calculated, and the difference value is equal to or greater than a threshold value for motion recognition (greater than a predetermined value). If the difference value is equal to or greater than the threshold value for motion recognition, it is recognized that there has been motion.

  The input image may be an RGB image or a reflected light image. For example, a difference value between the total value of the color values of each pixel of the RGB image at time T20 and the total value of the color values of each pixel of the RGB image at time T21 is calculated, and whether or not the difference value is equal to or greater than a threshold value. Determine whether. Then, when the difference value is equal to or greater than the threshold value, it is recognized that there has been movement.

  Also, a difference value between the total depth value of each pixel of the reflected light image at time T20 and the total depth value of each pixel of the reflected light image at time T21 is calculated, and the difference value is equal to or greater than the threshold value. Determine whether or not. Then, when the difference value is equal to or greater than the threshold value, it is recognized that there has been movement. In particular, if a reflected light image is used, it can be recognized that the object has moved in the depth direction.

  In the present embodiment, since a specific area is set in the input image, a process of recognizing a change in the image in the specific area may be performed at a predetermined timing. For example, as shown in FIGS. 33A and 33B, the total value of the pixel values of the specific area A20 of the input image F20 acquired at time T20 and each of the specific areas A21 of the input image F21 acquired at time T21. A difference value of the total value of the pixel values may be calculated, and when the difference value is equal to or greater than a threshold value, it may be recognized that there is movement. This is because the movement of the object can be recognized efficiently, the machine power can be poured into the specific area, and the movement of the object can be recognized more accurately.

  For example, a difference value between the total color value of each pixel in the specific area A20 and the total color value of each pixel in the specific area A21 is calculated, and it is determined whether the difference value is equal to or greater than a threshold value. . Then, when the difference value is equal to or greater than the threshold value, it is recognized that there has been movement. Also, a difference value between the total depth value of each pixel in the specific area A20 and the total depth value of each pixel in the specific area A21 is calculated, and it is determined whether or not the difference value is equal to or greater than a threshold value. . Then, when the difference value is equal to or greater than the threshold value, it is recognized that there has been movement.

Object recognition device 10, processing unit 100, acquisition unit 110, calculation unit 111,
Area setting unit 112, object recognition processing unit 113, game calculation unit 114,
Image generation unit 120, sound control unit 130, storage unit 170, main storage unit 171,
Drawing buffer 172, recognition pattern storage unit 173, input image storage unit 174,
Difference image storage unit 175, information storage medium 180, communication unit 196,
Input unit 20, RGB camera (imaging unit) 21, processing unit 22, storage unit 23,
Object recognition device 50, processing unit 500, acquisition unit 510, calculation unit 511,
Area setting unit 512, object recognition processing unit 513, game calculation unit 514,
Image generation unit 520, sound control unit 530, storage unit 570, main storage unit 571,
Drawing buffer 572, recognition pattern storage unit 573, input image storage unit 574,
Difference image storage unit 575, information storage medium 580, communication unit 596,
Input unit 60, light emitting unit 610, light source 611, depth sensor 620,
RGB camera (imaging unit) 630, sound input unit 640, processing unit 650,
Storage unit 660 display unit 90, speaker 92, P player,
F1-F4, F10, F11 input image, P1, P2 pixels, V motion vector,
A1, A2, B1, C1, C2 specific area, S1, S1 ′, S2, S2 ′ movement area,
GP input unit position, Q object position, L object to input unit distance,
BO1, BO2, BO3, BO4, BO5, BOH1 bone,
ST1, ST2, ST2 ', ST3, ST3' silhouette

Claims (16)

A program for performing object recognition processing,
An input image having a depth value of each pixel is obtained by irradiating the object with light and receiving reflected light of the object, and a specific region is set in the input image based on the depth value of each pixel of the input image An area setting section;
As an object recognition processing unit that performs object recognition processing for recognizing an object, the computer functions,
The object recognition processing unit
A program characterized by performing object recognition processing in a specific area. In claim 1,
The object recognition processing unit
A program for performing object recognition processing in a specific area by increasing the accuracy of object recognition processing in a specific area higher than the accuracy of object recognition processing in an area other than the specific area. In claim 1 or 2,
The object recognition processing unit
A program for performing object recognition processing in a specific area by setting a period for performing object recognition processing in a specific area to be shorter than a period of object recognition processing in an area other than the specific area. In any one of Claims 1-3,
The object recognition processing unit
A program for performing object recognition processing in a specific area by increasing the image accuracy of the specific area to be higher than that of an area other than the specific area. In any one of Claims 1-4,
The object recognition processing unit
While performing object recognition processing in areas other than the specific area,
A program characterized in that a period for performing object recognition processing in an area other than the specific area is longer than a period for performing object recognition processing in the specific area. In any one of Claims 1-5,
The object recognition processing unit
While performing object recognition processing in areas other than the specific area,
A program characterized in that the image accuracy of a region other than the specific region is lower than the image accuracy of the specific region. In any one of Claims 1-6,
When the area setting unit sets a plurality of specific areas,
The object recognition processing unit
A program for performing object recognition processing on at least one specific area. In claim 7,
The area setting unit gives priority to each specific area,
The object recognition processing unit
While performing object recognition processing in each specific area,
A program characterized in that a period for performing object recognition processing in a specific area with low priority is made longer than a period for performing object recognition processing in a specific area with high priority. In claim 7 or 8,
The area setting unit gives priority to each specific area,
The object recognition processing unit
While performing object recognition processing in each specific area,
A program characterized in that the image accuracy of a specific region with a low priority is made lower than the image accuracy of a specific region with a high priority. In claim 7,
The area setting unit gives priority to each specific area,
The object recognition processing unit
A program that performs object recognition processing in a specific area with high priority without performing object recognition processing in the specific area with low priority. In any one of Claims 1-10,
The region setting unit
A program that sets a specific region in an input image based on color information and depth value of each pixel of the input image. In any one of Claims 1-11,
The region setting unit
A program that sets a specific region in an input image based on a depth value that is equal to or greater than a predetermined value among the depth values of each pixel of the input image. In any one of Claims 1-12,
The object recognition processing unit
A program that performs the object recognition processing in the specific area based on bone information. In any one of Claims 1-13,
The object recognition processing unit
A program for performing a process of recognizing a change in an image in the specific area at a predetermined timing.   An information storage medium readable by a computer, wherein the program according to any one of claims 1 to 14 is stored. An object recognition system that performs processing for recognizing an object,
An input image having a depth value of each pixel is obtained by irradiating the object with light and receiving reflected light of the object, and a specific region is set in the input image based on the depth value of each pixel of the input image An area setting section;
An object recognition processing unit for performing object recognition processing for recognizing an object,
The object recognition processing unit
An object recognition system characterized by performing object recognition processing in a specific area.