JP2011027707A - Person motion detecting device, play apparatus, person motion detecting method, gaming method, control program, and readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect a direction and a change amount of range of a person motion in an image capturing range without making a computation amount to be huge even in a high frame rate when many space areas are planarly computed. <P>SOLUTION: A high speed person motion detection control means 4 is adapted to detect motion information of a person on the basis of range information from a TOF range image sensor 31. The high speed person motion detection control means 4 includes a range frame generating unit 431 that generates range frames in a time series manner on the basis of the range information from the TOF range image sensor 31, a range balance frame generating unit 432 that generates a range balance frame (background image frame) as range information not having a range change of a person on the basis of at least two range frames, and a frame difference computing section 46 that computes a frame difference between each of the plurality of range frames and the range balance frame by each range frame, and computes a change amount of a range and a change direction of the person as motion information of the person based on the frame difference. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a human motion detection device and a human motion detection method for detecting a human motion from a distance frame and extracting target personal information, and to detect a human motion using the human motion detection device to Play equipment such as a video game machine that performs various controls of the above, a game method using the same, a control method for causing a computer to execute the person motion detection method and the game method, and a computer-readable computer storing the control program The present invention relates to a readable recording medium.

  2. Description of the Related Art Conventionally, for playground equipment such as a video game machine, many experience-type games using human movement have been developed on sports such as tennis and baseball. In these experience-type games, the controller is regarded as a tennis racket or a baseball bat, for example, and the controller such as the racket or bat is moved by swinging or protruding in accordance with the motion of the animation on the television screen. Changes in the speed and direction of the controller's movement with time are detected by an acceleration sensor and angular velocity sensor installed in the controller, and the movement of the tennis racket and baseball bat on which the characters in the game operate are displayed on the television screen. Furthermore, it is reflected in the occurrence of various events.

  A conventional playground equipment using a controller will be described with reference to FIG.

  FIG. 29 is an external view of a conventional playground equipment.

  In FIG. 29, a conventional playground equipment 111 includes a home-use television receiver 113 (hereinafter referred to as a monitor) as an example of a display means, and a stationary game apparatus main body 112 connected to the monitor 113 via a connection cord. The game apparatus main body 112 is configured by a game controller 114 that is connected wirelessly. The game controller 114 includes an acceleration sensor 115. The acceleration sensor 115 detects a change in the speed and direction of movement of the controller as described above, and transmits the detected value to the game apparatus body 112 wirelessly. ing.

  However, the sensor can detect only the movement of the controller moved by a human, and cannot detect the position or movement of the entire human body in the three-dimensional space. For this reason, the game operation is oversimplified, and there is a problem that the user cannot obtain a sufficient operation feeling or gets used to the game operation immediately and gets bored.

  For example, in a tennis game, a strong hitting ball can be obtained simply by moving a wrist having a controller as a racket, so that the feeling of operation is different from that of actual tennis using the whole body. For example, in a baseball game, the position of the controller as a bat, such as the height in the three-dimensional space, is not detected, so the height of the bat swing cannot be distinguished, and only the timing at which the bat and the ball collide is monitored.

  Therefore, if it is possible to detect the position and movement of a human in a three-dimensional space and reflect it in the game content, it is possible to provide a game with a high sense of realism with a sense of operation close to that of real sports, or through the game It can be expected to improve technology in sports.

  However, in order to detect the position and movement of a human in a three-dimensional space, a conventional RGB camera is used to analyze the differences in the RGB image between the target person and the background, and extract the target object (For example, using a color difference from a single color background to cut out the target person) requires a special imaging environment or complicated image recognition processing, so it is widely used. It has been difficult to apply to play equipment such as a video game machine.

  Therefore, it is expected to apply a distance image sensor for the above purpose. If this distance image sensor is used, it is possible to detect only the target object within the set distance range, so it is possible to detect the position and motion of a person without using a special background or image recognition processing. .

  In particular, in the experience-type game such as sports, the motion of the target person is fast, and therefore the frame rate of the distance image sensor needs to be increased in order to improve the user's operational feeling and motion detection accuracy.

  However, when the frame rate is increased, the signal accumulation time of a solid-state imaging device such as a CCD image sensor or a CMOS image sensor is reduced, so that the S / N ratio is lowered. However, if complicated signal processing such as advanced filtering processing is performed to distinguish the signal component from the noise component, the amount of calculation increases, and real-time processing according to the progress of the game becomes difficult, or In order to perform real-time processing, high-performance hardware is required, which increases the cost of parts.

  For this reason, in particular, when detecting a human action with a playground equipment such as a video game machine at a high frame rate, it is required to perform processing in real time based on computationally inexpensive calculation means.

  A conventional information extraction apparatus of this type using a distance frame is disclosed in Patent Document 1. In the conventional information extraction apparatus, the distance to the target object is acquired from the intensity distribution of the reflected light. That is, since the intensity of the reflected light decreases in inverse proportion to the square of the distance to the target object, the perspective relationship of each component of the target object in the imaging range is obtained from the intensity distribution of the reflected light.

  FIG. 17 is a block diagram illustrating a configuration example of a main part of a conventional information extraction device disclosed in Patent Document 1 using a distance frame. In FIG. 17, information such as the shape, movement, and position of an object is extracted (recognized) from a plurality of locations in one imaging region, and the extracted information is used as another system such as a mole hitting game. Is input.

  As shown in FIG. 17, a conventional information extraction apparatus 100 receives reflected light and acquires a distance image acquisition unit 101 that acquires a distance image, and an area division processing unit that divides the acquired distance image into separately designated areas. 102, and a distance image is analyzed for each region divided by the region division processing unit 102, and the contour of the object, the extraction of the center of gravity, the calculation of the distance to the object, the movement speed of the object, and the calculation of the movement vector An image processing unit 103 that extracts target information related to a recognition target, a presentation unit 104 including, for example, a large liquid crystal panel or a speaker, and an information management unit 105 that controls each unit. .

  Information extracted by the image processing unit 103 is input to another system via the information management unit 105, for example. That is, as another system, for example, a mole hitting game is connected to the other system via the information management unit 5. The other system may be a game system equipped with a control program that can be executed by a computer, for example.

  The information extraction apparatus 100 as shown in FIG. 17 causes a mole to appear from an arbitrary display area on the display screen of the presenting unit 104, for example, and the user pulls toward the display area where the mole appears (the user moves to the display screen). If the player is connected to a mole hitting game that competes for the number of moles hit, the one distance image acquisition unit 101 can select which mole on the display screen of the presenting unit 104. It is installed so that it can recognize whether it is going to be pulled (so that the position of the hand as an object relative to the entire display screen can be recognized).

  The area division processing unit 102 divides the area of the distance image acquired by the distance image acquisition unit 101 corresponding to the mole appearance area on the display screen of the presentation unit 104.

  Here, the distance image acquisition unit 101 and the distance image acquired by the distance image acquisition unit 101 will be briefly described.

  The appearance of the distance image acquisition unit 101 is shown in FIG. As shown in FIG. 18, a light receiving unit 101a composed of a circular lens and an area sensor (not shown) in the rear part is arranged at the center, and infrared light or the like is provided around the circular lens along its contour. A plurality of (e.g., eight) light emitting portions 101b each composed of LEDs that irradiate the light are arranged at equal intervals.

  The light emitted from the light emitting unit 101b is reflected by the object, collected by the lens of the light receiving unit 101a, and received by the area sensor at the rear of the lens. This area sensor is, for example, a plurality of sensor groups arranged in a 256 × 256 matrix, and the intensity of reflected light received by each sensor in the matrix is a pixel value. The image acquired in this way is distance image data as the intensity distribution of reflected light for each divided region as shown in FIG.

  FIG. 20 is a block diagram illustrating a configuration example of a main part of the distance image acquisition unit 101 in FIG. 17, which mainly includes a light receiving unit 101a, a light emitting unit 101b, a reflected light extraction unit 101c, and a timing signal generation unit 101d. ing.

  The light emitting unit 101b emits light whose intensity varies with time according to the timing signal generated by the timing signal generating unit 101d. This light is applied to the target object in front of the light emitting unit 101b.

  The light receiving unit 101a detects the amount of light reflected by the target object of the light emitted from the light emitting unit 101b.

  The reflected light extraction unit 101c extracts a spatial intensity distribution of the reflected light received by the light receiving unit 101a. Since the spatial intensity distribution of the reflected light can be captured as an image, this is hereinafter referred to as a distance image.

  The light receiving unit 101a generally receives not only light reflected from an object of light emitted from the light emitting unit 101b but also external light such as illumination light and sunlight. Therefore, the reflected light extraction unit 102 takes the difference between the amount of light received when the light emitting unit 101b emits light and the amount of light received when the light emitting unit 101b does not emit light, thereby obtaining the light emitting unit 101b. Only the reflected light component of the light from the target object can be extracted.

  The reflected light extraction unit 101c extracts the intensity distribution, that is, distance image data as shown in FIG. 19, from the reflected light received by the light receiving unit 101a.

  For the sake of simplicity, FIG. 19 shows a case of an 8 × 8 pixel distance image that is a part of a 256 × 256 pixel distance image.

  The reflected light from the object decreases significantly as the distance of the object increases. When the surface of the object scatters light uniformly, the amount of light received per pixel in the distance image decreases in inverse proportion to the square of the distance to the object.

  In FIG. 19, the value of the cell (pixel value) in the matrix indicates the intensity of the acquired reflected light in 256 gradations (8 bits). For example, a cell having a value of “255” is the state closest to the distance image acquisition unit 101, a cell having a value of “0” is far from the distance image acquisition unit 101, and reflected light is a distance image acquisition unit. It is shown that 101 is not reached.

  Each pixel value of the distance image represents the amount of reflected light received by the unit light receiving unit corresponding to the pixel. Reflected light is affected by the nature of the object (specularly reflects, scatters, absorbs, etc.), the direction of the object, the distance of the object, etc., but the entire object scatters light uniformly. In this case, the amount of reflected light is closely related to the distance to the object. Since the hand and the like have such properties, the distance image when the hand is put out in front of the distance image acquisition unit 101 reflects the distance to the hand, the inclination of the hand (partly different distance), and the like. A three-dimensional image as shown in FIG. 21 can be obtained.

JP 2004-333505 A

  However, the conventional information extraction apparatus 100 disclosed in Patent Document 1 can only obtain the relative value of the distance of each component of the target object, and cannot obtain the absolute value of the distance. When it is necessary to obtain the absolute value of the distance, a calibration procedure for obtaining a reference correspondence between the distance and the reflected light intensity is required, or a procedure for converting from a relative value to an absolute value is required. .

  Further, in the conventional information extraction device 100 disclosed in Patent Document 1, since the distance is obtained based on the intensity of reflected light, the entire surface of the target object can be applied only to an object having uniform optical properties. In general, an object has different optical properties (for example, reflection, scattering, absorption, and refraction) depending on the surface material, color, processing, and the like. When a human is a target object, the human is composed of a plurality of components having different optical properties (for example, hair, skin, upper body clothes, and lower body clothes). When acquiring distance information of a target composed of a plurality of components having different optical properties, it becomes necessary to perform a distance correction process taking into account the optical properties of each component. When applied to a playground equipment such as a sports game, the amount of calculation is further increased while the need to drive the distance image sensor at a high frame rate to perform calculation processing of a large number of distance frames, Real-time processing becomes difficult.

  Furthermore, in the conventional information extraction apparatus, the orientation of the target object with respect to the apparatus also affects the distance. Regardless of its orientation, if it is not an object that reflects and scatters light uniformly (eg, a sphere), it depends on the orientation of the target object relative to the device (eg, facing directly, facing diagonally, or facing sideways) Since the intensity of the reflected light is different, the distance information is also acquired differently. That is, even if the translational motion of the target object can be detected, the rotational motion cannot be accurately detected.

  The present invention solves the above-described conventional problem. When a large number of spatial regions are subjected to arithmetic processing, the amount of calculation does not become enormous even at a high frame rate, and the direction and distance of the human motion in the imaging range are not affected. Human motion detection device and human motion detection method capable of detecting change detection accuracy with high accuracy, play equipment such as a video game machine using the same, game method using the same, human motion detection method and game method using the same It is an object of the present invention to provide a control program for causing a computer to execute at least one of them, and a computer-readable readable recording medium in which the control program is stored.

  The human motion detection apparatus according to the present invention receives a plurality of light receiving means for emitting projection light and reflected light from the projection space of the projection light and receiving distance information according to the distance to the person in the projection space. A person motion detection device having a TOF (time of flight) range image sensor output from each unit, and detecting person motion information based on distance information from the TOF range image sensor It has a detection control means, and the above object is achieved thereby.

  Preferably, the human motion detection control unit in the human motion detection device of the present invention includes a distance frame generating unit that generates a plurality of distance frames in time series based on distance information from the TOF type distance image sensor, Frame difference calculation means for calculating each frame difference for a plurality of distance frames and calculating a change amount and a change direction of the distance of the person as motion information of the person from the frame difference.

  Further preferably, the human motion detection control means in the human motion detection device of the present invention comprises at least a distance frame generating means for generating a plurality of distance frames in time series based on distance information from the TOF type distance image sensor, and at least Based on the two distance frames, distance balanced frame generating means for generating a distance balanced frame that is distance information with no distance change for the person, and calculating a frame difference between each of the plurality of distance frames and the distance balanced frame. And a frame difference calculation means for calculating a change amount and a change direction of the distance of the person as the motion information of the person from the frame difference.

  Further preferably, in the human motion detection device of the present invention, the distance for measuring the distance to the person in real time and detecting the presence or absence of a change in the distance from the person from the frame difference between at least two distance frames Displacement detecting means is further included.

  Further preferably, in the human motion detection device of the present invention, the distance to the person is measured in real time, and the state where the frame difference between at least two distance frames is less than a predetermined threshold is greater than or equal to a predetermined time or predetermined When the number of frames continues, the distance balance state determination means for determining that the distance information of the person is in an equilibrium state is further included.

  Further preferably, the distance balanced frame generating means in the human motion detecting device of the present invention calculates a time average of at least two distance frames when the distance balanced state determining means determines that the distance balanced state is in the distance balanced state. Generate a distance balanced frame.

  Further preferably, the distance balanced frame generation means in the human motion detection device of the present invention moves the person to the right or left side of the imaging range according to an instruction from the play equipment when the person is within the imaging range. Generating a distance balanced frame in one half of the imaging range in the left or right half of the imaging range, with no person in the imaging range, and moving the person to the left or right side of the imaging range, to the right or left side of the imaging range A distance balanced frame is generated by generating a distance balanced frame of the other half that does not have the person in the half imaging range and synthesizes a distance balanced frame that does not have the person in the imaging range.

  Further preferably, in the human motion detection device of the present invention, a frame for calculating a frame difference between the distance frame and the distance balanced frame and detecting a person based on whether or not a distance from the background in which the distance information is in a balanced state is detected. It has a difference calculation means.

  Furthermore, it is preferable that the human motion detection apparatus of the present invention further includes a central coordinate detection unit that obtains a central coordinate of the person from distance information of the person detected by a frame difference between the distance frame and the distance balanced frame.

  Further preferably, the center coordinate detection means in the human motion detection device of the present invention obtains a movement amount of the center coordinates between the distance frames.

  Further preferably, the center coordinate detection means in the human motion detection device of the present invention is characterized in that the center coordinate obtained from the distance displacement obtained by the difference calculation between the distance frames and the frame difference between the distance frame and the distance balanced frame. From this, the relative position of the distance displacement and the center coordinate is calculated.

  Further preferably, the center coordinate detection means in the human motion detection device of the present invention calculates the center-of-gravity coordinates of each coordinate axis in the three-dimensional space as the center coordinates from the detected distance information of the whole body of the person.

  Further preferably, the center coordinate detection means in the human motion detection device of the present invention is configured to determine the center point from the maximum value and the minimum value in each coordinate axis direction of the three-dimensional space from the detected distance information of the whole body of the person. Calculate as

  Further preferably, the center coordinate detection means in the human motion detection device of the present invention is configured such that, based on the detected distance information of the entire body of the person, a point at which the sum of the distances from each point on the person's contour is minimum is the center. Calculate as coordinates.

  Further preferably, the distance balanced frame generating means in the human motion detecting apparatus of the present invention performs the distance balanced frame generation processing at the time of starting the apparatus, at the time of initialization, or every predetermined time.

  Further preferably, the light emitting means in the human motion detection device of the present invention projects near-infrared pulse light as the projection light into the projection space at a constant period.

  Further preferably, the light receiving unit in the human motion detecting device of the present invention has a photoelectric conversion unit provided in the center, and two charges to which mutually opposite gate signals are input at a point-symmetrical position with respect to the center of the photoelectric conversion unit. Two pixel output electrodes V1 and V2 are provided via transfer gates, respectively, and signal charges distributed depending on the flight time of light corresponding to the distance from the two pixel output electrodes V1 and V2 to the person. Are output respectively.

  Further preferably, the signal charges from the two pixel output electrodes V1 and V2 in the human motion detection device of the present invention are converted into output voltage information corresponding to the distance to the person, respectively, and the TOF distance image sensor To output the distance information.

  Further, preferably, the distance L to the person in the human motion detection device of the present invention detects a flight time T of light to the person with a known speed of light as c, and L = (1/2) · c · Required from T.

  Still preferably, the distance information in the human motion detection device of the present invention is three-dimensional distance information for each pixel corresponding to the light receiving unit.

  The playground equipment of the present invention uses the person motion detection device of the present invention, the amount of change in the distance of the person from the person motion detection device, and the motion information of the direction of the change, and the actions and events of the characters in the game. And an application processor that controls the game so as to be reflected in the occurrence of the above, thereby achieving the above object.

  Preferably, in the playground equipment of the present invention, the application processor compares the frame difference information obtained continuously in order to obtain detailed information of the motion of the person, and the motion area of the person. Human motion detection means for extracting various information of the position information of the person, the speed vector of the motion of the person, and the acceleration vector of the motion.

  Further preferably, in the playground equipment of the present invention, the position information of the motion area is obtained three-dimensionally from the XY coordinate positions in the horizontal and vertical directions of the distance frame and the distance L from the playground equipment, and the motion of the person These velocity vectors and acceleration vectors are obtained from any one of the average moving distance of the motion area, the movement distance of the center of gravity position of the motion area, and the movement distance of the specific part position of the person.

  Furthermore, it is preferable that the playground equipment of the present invention is limited to a distance frame that has an expected value with respect to the time and space area of the action performed by the person, and includes time and space distance information that matches the expected value. Then, the processing by the person motion detection means is performed.

  Further preferably, when the movement amount of the central coordinate of the person in the playground equipment of the present invention is equal to or greater than the threshold set from the expected value, it is determined that the person has “whole body movement”, and There is a center coordinate movement determination means for determining that the person has not moved the whole body when the movement amount of the center coordinates of the person is less than the threshold.

  Further preferably, the playground equipment according to the present invention further comprises body part determination means for determining the body part of the operated person from the relative positional relationship of the distance displacement of the difference between the center coordinate and the distance frame.

  Further preferably, the body part determination means in the playground equipment of the present invention is such that if the distance displacement detected by the frame difference between the distance frames is in the upper right from the center coordinate, the body part determination means is in the right arm operation and in the upper left from the center coordinate. If it is a left arm motion, if it is lower right than the center coordinate, it is determined that the motion is a right foot, and if it is lower left than the center coordinate, it is determined that the motion is a left foot.

  Further preferably, the body part determination means in the playground equipment of the present invention compares the expected value with at least one of a time, a spatial region, and a motion change when the body part exists as a determination result. If any of the body parts agrees with the expected value, it is determined that the body part has “expected motion”, and if any of the body parts does not match the expected value, the body part has “no expected action”. It is determined.

  Furthermore, preferably, as an expected value in the playground equipment of the present invention, the expected time is the timing at which the person is expected to take some action, and there is an expected space area in which the person is expected to move It is assumed that the speed and direction of the motion that the person is expected to perform are the expected motion changes.

  Further preferably, the application processor in the playground equipment according to the present invention has an expected value related to at least one of the time of the person's action, the space region, and the action change, and the person action that matches the expected value In the case of detecting sensation, the apparatus has a sensation generating device that generates at least one of vibration, light, sound and scent via wireless or wired communication.

  Further preferably, in the playground equipment of the present invention, the human motion is detected using the human motion detection device without using a game controller mounted on a conventional playground equipment, and the detection result of the human motion and the A comparison process with the expected value is performed to drive the sensation generating device that generates at least one of vibration, light, voice, and scent.

  In the human motion detection method of the present invention, reflected light from the projection space of the projection light emitted from the light emitting means is received, and distance information according to the distance to the person in the projection space is expressed as TOF (Time of Flight). ) Method for detecting human motions respectively output from a plurality of light receiving sections of a distance image sensor, comprising a human motion detection control step for detecting human motion information based on distance information from the TOF distance image sensor Thus, the above object is achieved.

  Preferably, the human motion detection control step in the human motion detection method of the present invention includes a distance frame generation step of generating a plurality of distance frames in time series based on distance information from the TOF distance image sensor, A frame difference calculation step of calculating each frame difference for a plurality of distance frames and calculating a change amount and a change direction of the distance of the person as motion information of the person from the frame difference.

  Further preferably, the human motion detection control step in the human motion detection method of the present invention includes a distance frame generation step of generating a plurality of distance frames in time series based on distance information from the TOF distance image sensor, and at least Based on the two distance frames, a distance balanced frame generation step for generating a distance balanced frame that is distance information without any change in the distance of the person, and a frame difference between the plurality of distance frames and the distance balanced frame is calculated. And a frame difference calculation step for calculating a change amount and a change direction of the distance of the person as the motion information of the person from the frame difference.

  Further preferably, in the human motion detection method of the present invention, the distance displacement detection for measuring the distance of the person in real time and detecting the change in the distance of the person from the frame difference between at least two distance frames. It further has a step.

  Further preferably, in the human motion detection method of the present invention, the distance of the person is measured in real time, and the state where the frame difference between at least two distance frames is less than a predetermined threshold is greater than or equal to a predetermined time or predetermined The method further includes a distance equilibrium state determination step for determining that the distance information of the person is in an equilibrium state when the number of frames continues.

  Further preferably, in the human motion detection method of the present invention, a frame difference between the distance frame and the distance balanced frame is calculated, and the person is detected based on whether there is a change in the distance from the background where the distance information is in a balanced state. A frame difference calculation step.

  Furthermore, it preferably has a center coordinate detection step of obtaining center coordinates from distance information of a person detected by a frame difference between the distance frame and the distance balanced frame in the person motion detection method of the present invention.

  Still preferably, in a person motion detection method according to the present invention, the center coordinate detection step obtains a movement amount of the center coordinates between the distance frames.

  Further preferably, the center coordinate detection step in the human motion detection method of the present invention includes the center coordinate obtained from the distance displacement obtained by the difference calculation between the distance frames and the frame difference between the distance frame and the distance balanced frame. From this, the relative position of the distance displacement and the center coordinate is calculated.

  Still preferably, in the person motion detection method of the present invention, the center coordinate detection step calculates the center-of-gravity coordinates of each coordinate axis in the three-dimensional space as the center coordinates from the detected distance information of the whole body of the person.

  Further preferably, the central coordinate detection step in the human motion detection method of the present invention includes determining the central point from the maximum value and the minimum value in each coordinate axis direction of the three-dimensional space from the distance information of the detected human whole body. It is calculated as follows.

  Further preferably, the center coordinate detection step in the person motion detection method of the present invention is configured such that the center of the distance from each point on the contour of the person is the minimum from the detected distance information of the whole body of the person. Calculate as coordinates.

  In the game method of the present invention, the application processor uses the amount of change in the distance of the person from the person motion detection device of the present invention and the motion information of the change direction to reflect the motion of the character in the game and the occurrence of the event. A game execution step for controlling the game so that the above-mentioned object is achieved.

  Preferably, in the game method of the present invention, in order to acquire the detailed motion information of the person, the frame difference information acquired continuously is compared, the positional information of the motion area of the person, the motion of the person A person motion detection step for extracting various information of the velocity vector and acceleration vector of the human body, and information on the distance of the person and the direction of the change, as well as the position information of the person's motion region and the motion of the person Using the various information on the velocity vector and the acceleration vector, as the game execution step, the application processor controls the game so that it is reflected in the actions of the characters in the game and the occurrence of events.

  The control program according to the present invention describes a processing procedure for causing a computer to execute the steps of the human motion detection method according to the present invention, thereby achieving the above object.

  The control program according to the present invention describes a processing procedure for causing a computer to execute the steps of the game method according to the present invention, thereby achieving the above object.

  The readable recording medium of the present invention is a computer-readable medium in which the control program of the present invention is stored, whereby the above object is achieved.

  With the above configuration, the operation of the present invention will be described below.

  In the present invention, the distance frame generating means for generating a plurality of distance frames in time series based on the distance information from the TOF type distance image sensor and the frame having no distance change based on at least two distance frames. A distance balanced frame generating means for generating a distance balanced frame; and a means for calculating a frame difference between the distance balanced frame and the distance balanced frame for each distance frame, and calculating a distance change amount and a direction of the target object. The direction and distance to the distance displacement region whose shape is specified as a person is detected as the motion information of the person.

  As a result, even when a large number of spatial regions are processed in a plane, the amount of calculation does not become enormous, unlike conventional distance image sensors and triangulation distance image sensors that use the intensity distribution of reflected light. Even when the frame rate is high, it is possible to detect the movement of a person in real time.

  Therefore, the high-speed detection of the person action enables the game content to be controlled in real time according to the person action, and greatly improves the operability and the motion detection accuracy in the playground equipment.

  As described above, according to the present invention, the distance displacement detection means for detecting the distance displacement of the projection light to the object in the projection space based on the distance information (output voltage information corresponding to the distance) from the TOF type distance image sensor. A distance equilibrium state determination means for determining the presence / absence of a distance displacement, in order to detect a distance displacement due to a frame difference between the distance frame and the distance equilibrium frame as a human motion, and when calculating a large number of spatial regions in a plane Even so, the amount of calculation does not become enormous as in the case of the conventional distance image sensor, and the detection of the movement of the person can be performed computationally at low cost. Therefore, even in applications where the required frame rate is high, it is possible to extract a person's action in real time and reflect it in the game content and progress.

It is a block diagram which shows the principal part structural example of the person motion detection apparatus which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows typically the TOF type | formula distance measurement principle used with the person motion detection apparatus of FIG. FIG. 2 is a plan structure diagram of a light receiving unit in one pixel unit in the pixel array unit of FIG. 1. FIG. 3 is a timing diagram of signal waveforms at each main part for explaining the principle of TOF distance measurement in FIG. 2. It is a block diagram which shows the principal part structural example of the playground equipment apparatus which concerns on Embodiment 2 of this invention. It is a figure which shows typically the expected time and expected presence space in the best form for implementing this Embodiment 2. FIG. It is a figure which shows typically the expected time in the best form for implementing this Embodiment 2, an expected presence space, and an expected operation | movement change. It is a flowchart which shows the simple operation example of the person motion detection apparatus of FIG. It is a flowchart which shows the example of a highly accurate operation | movement of the person motion detection apparatus of FIG. It is the image figure which showed the specific example of the distance information acquired by the distance balance frame production | generation part of FIG. It is the image figure which showed the specific example of the distance information acquired by the distance frame production | generation part of FIG. 5 at a certain time. FIG. 11 is an image diagram illustrating a specific example of distance information acquired by the distance frame generation unit in FIG. 5 at a time different from that in FIG. 10. FIG. 12 is an image diagram illustrating a specific example of a frame difference image (RFsubBRF) between the distance frame of FIG. 10 and the distance balanced frame of FIG. 11. 12 is an image diagram showing a specific example of a frame difference image (RFsubBRF) between the distance frame of FIG. 10 and the distance balanced frame of FIG. 11. FIG. 13 is an image diagram illustrating a specific example of a frame difference image between the BRF distance difference frame (RFsubBRF) in FIG. 12 and the BRF distance difference frame (RFsubBRF) in FIG. 11. FIG. 6 is a block diagram illustrating a hardware configuration example of a main part of the application processor of FIG. 5. It is a block diagram which shows the principal part structural example of the conventional information extraction apparatus currently disclosed by patent document 1 using a distance frame. It is a figure which shows typically the external appearance of the distance image acquisition part in the conventional information extraction device of FIG. FIG. 18 is a data diagram of a distance image showing an intensity distribution of reflected light for each divided region in the conventional information extraction device of FIG. 17. It is the block diagram which showed the principal part structural example of the distance image acquisition part in the conventional information extraction device of FIG. It is a figure which shows the distance image at the time of pushing out a hand ahead of the distance image acquisition part in the conventional information extraction device of FIG. It is a block diagram which shows the principal part structural example of the playground equipment apparatus which concerns on Embodiment 3 of this invention. It is a flowchart which shows the operation example of the person motion detection apparatus of FIG. It is a block diagram which shows the principal part structural example of the playground equipment apparatus which concerns on Embodiment 4 of this invention. It is a flowchart which shows the operation example of the person motion detection apparatus of FIG. It is a block diagram which shows the principal part structural example of the playground equipment apparatus which concerns on Embodiment 5 of this invention. It is a block diagram which shows the modification of the playground equipment apparatus which concerns on Embodiment 5 of this invention. It is a block diagram which shows another modification of the playground equipment apparatus which concerns on Embodiment 5 of this invention. It is an external view of the conventional playground equipment apparatus.

  Hereinafter, after describing the first embodiment of the human motion detection device of the present invention, the first embodiment of the human motion detection device of the present invention controls the content and progress of the game according to the motion content of the person as the target object. Embodiment 2 is used for a playground equipment apparatus, and a method for determining the movement of the entire body of a person in detection of human motion is described in Embodiment 3. In the detection of human motion, the distance displacement between the distance frames is determined by which part of the body. A method for determining whether or not this is the case will be described as a fourth embodiment, and a playground equipment having a sensation generating device will be described as a fifth embodiment in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration example of a main part of the human motion detection device according to the first embodiment of the present invention.

  In FIG. 1, as shown in FIG. 2, a human motion detection apparatus 1 according to the first embodiment includes a light emitting means 2 as a light source for emitting projection light to a target object A for distance measurement, and the projection light. A light receiving means 3 for receiving reflected light from the target object A for distance measurement, and a human motion detection control means 4 for detecting the motion of the person as the target object A based on the light reception information from the light receiving means 3. TOF (determining the distance L to the target object A by calculation from the known speed of light c by detecting the flight time (delay time) T of the light to the target object A based on the light reception information from the light receiving means 3 Using a time-of-flight technique, the position and motion of a person are detected with high accuracy in three dimensions using distance information of a plurality of points for each pixel.

  The light emitting means 2 includes a power LED driver 21 whose emission timing of projection light is controlled by the human motion detection control means 4, and a power LED module 22 that is driven by the power LED driver 21 to emit projection light from a plurality of power LEDs 22a. And a radiation angle adjusting lens 23 which is disposed in front of the power LED module 22 provided with at least one power LED 22a and which can adjust the radiation angle of the projection light with respect to the target object A.

  The projection light in this case may project near-infrared pulsed light as projection light onto the projection space at a constant period.

  The light receiving means 3 receives the reflected light from the target object A and detects an imaging signal corresponding to the distance L to the target object A with a plurality of pixels, and a TOF distance image sensor 31. And a viewing angle adjustment lens 32 that can adjust the viewing angle at which the reflected light from the target object A is incident.

  The TOF type distance image sensor 31 includes a pixel array unit 311 that is a plurality of light receiving units, a timing generation unit 312 that generates a timing signal for signal readout from each pixel (light receiving unit), and each pixel (light receiving unit). The vertical scanner unit 313 that scans in the vertical direction, the horizontal scanner unit 314 that scans each pixel (light-receiving unit) in the horizontal direction, and the vertical scanner unit 313 and the horizontal scanner unit 314 are controlled by using this timing signal to control each pixel. From the pixel driver unit 315 that drives signal readout from the (light receiving unit), the noise canceller unit 316 that performs noise cancellation processing on the image signal read from the pixel array unit 311 by the pixel driver unit 315, and the noise canceller unit 315 And an A / D converter 317 for A / D converting and outputting a signal.

  In the pixel array unit 311, a plurality of light receiving units (for example, about 80,000 pixels) that receive reflected light from the target object A and photoelectrically convert them to generate signal charges, respectively, are arranged in a matrix in a matrix direction in a planar shape. However, here, the light receiving portion of one pixel unit having the characteristic configuration of the present invention is shown in the plan view of FIG.

  As shown in FIG. 3, the light receiving unit of one pixel unit includes a light receiving unit that receives reflected light from the target object A as a photoelectric conversion region (charge conversion region PG) in the center, and this charge conversion. Output electrodes V1 and V2 are provided on the left and right positions of the region PG via charge transfer gates TX1 and TX2, respectively, and signal charges read from the output electrodes V1 and V2 are timely transferred to the charge transfer gates TX1 and TX2. It is sorted by the reciprocal gate signal. In short, the signal charges distributed depending on the flight time of light (the reflected light delay time Td) corresponding to the distance L to the target object A are output from the two pixel output electrodes V1 and V2, respectively. Are converted into output voltage information corresponding to the distance L, and output from the TOF distance image sensor 31 as distance information. That is, as shown in FIG. 4, the pulse width T0 of the projection light is distributed as output voltages ΔV1 and ΔV2 corresponding to the distance L from the two pixel output electrodes V1 and V2 according to the timing of the reflected light delay time Td. Is output. Note that CD is a background light signal discharge gate provided at the upper and lower positions of the charge conversion region PG.

  Here, the principle of obtaining the distance L to the target object A in a three-dimensional manner using the TOF (Time of Flight) technique shown in FIG. 2 will be briefly described.

  First, assuming that the current is I, the charge Q is obtained as a time integral of time ΔT, and is given by Q = I · ΔT. The charge Q that can be accumulated by applying the voltage ΔV to the electrode is given by Q = C · ΔV, where C is the capacitance of the dielectric. From these two equations, the output terminal voltage V is given by ΔV = (1 / C) · I · ΔT. Using the output terminal voltage V, the output terminal voltages ΔV1 and ΔV2 are derived below based on the timing diagram of the pulse projection light and the signal charge transfer gate shown in FIG. As shown in FIG. 4, assuming that the lighting time width of the pulse projection light is T0, the photocurrent flowing by photoelectric conversion of the reflected light is Iph, the speed of light is c, and the output terminal capacitances are C1 and C2, the output terminal voltage V1, V2 can be obtained by the following equation 1 (Equation 1).

Further, as shown in FIG. 3, since the two output terminals have the same structure, the terminal capacities C1 and C2 are equal. That is, C1 = C2. Substituting this condition into Equation 1 (Equation 1) to obtain the distance L, the following Equation 2 (Equation 2) can be obtained.

The human motion detection device 1 using the TOF type distance image sensor 31 based on the TOF technology is a human motion detection device using a distance image sensor that acquires distance information using the intensity distribution of reflected light from a conventional target object. Compared with the case of, the following points are significantly superior.

  First, the distance L at a plurality of points in space can be measured not by the relative perspective of the distance of each pixel but by the absolute value of the distance. Since the distance L is obtained as an absolute value, signal processing after A / D conversion (for example, setting of a threshold value for determining the presence or absence of distance displacement) is easy. Therefore, no additional arithmetic processing is required, which is suitable for real-time processing in applications where a high frame rate is required such as a video game.

  On the other hand, in the conventional distance image sensor that obtains the distance from the intensity distribution of the reflected light, the distance L can be obtained only as a relative value, and therefore an operation for converting the distance L from the relative value to the absolute value is required. The amount will increase. Therefore, it is not suitable for real-time processing at a high frame rate in a playground equipment such as a video game.

  Second, since it is difficult to be affected by the optical properties of the surface of the target object A, the range of application of the target object A from which distance information can be acquired is wide. According to the above equation 2, the distance L is determined by the ratio of the output terminal voltages ΔV1 and ΔV2, and thus is not easily affected by the optical properties of the surface of the target object A. That is, when the optical properties of the surface of the target object A vary depending on the constituent parts of the target object A (for example, a part of the surface of the target object A has a high reflectance and the other parts have a low reflectance), ΔV1 and ΔV2 Although the absolute value increases or decreases depending on the above components, since the ratio of ΔV1 and ΔV2 is constant, the distance L is not easily affected by the optical properties of the surface of the target object A. This eliminates the need for additional arithmetic processing and is suitable for real-time processing at a high frame rate in a playground equipment such as a video game.

  On the other hand, in the conventional distance image sensor that obtains the distance L from the intensity distribution of the reflected light, the intensity distribution of the reflected light is greatly influenced by the optical properties of the surface of the target object A, so that the target has uniform optical properties. Applicable only to objects. For this reason, when trying to acquire the whole body motion of a person, arithmetic processing for viewing the amount of change for each component part with uniform optical properties (for example, only the hand, only the upper body clothes, only the lower body clothes) An arithmetic process for correcting the distance in consideration of each optical property is required. As a result, the amount of additional computation becomes enormous, making it unsuitable for real-time processing at a high frame rate such as a video game.

Returning to the description of FIG. 1, the high-speed human motion detection control means 4, which is a characteristic configuration of the first embodiment, includes various control units necessary for controlling the entire human motion detection device 1, and includes a power LED. A timing control unit 41 as a timing control unit for controlling each driving timing of the driver 21 and the TOF type distance image sensor 31, and a host interface unit as a host interface unit for communicating with a system such as the host-side playground equipment 11 42, a data formatting unit 43 as data formatting means for converting an output data stream based on distance information (output voltage information corresponding to the distance) from the TOF type distance image sensor 31 to a predetermined format, Distance as distance displacement detection means for detecting distance displacement in projection space A position detection unit 44; a distance equilibrium state determination unit 45 as a distance displacement determination unit that determines presence or absence of a distance displacement; and a frame difference calculation unit 46 as a frame difference calculation unit that obtains a frame difference between the distance frame and the distance balance frame. And detecting the motion of the person as the target object A based on the light reception information from the light receiving means 3.
The timing control unit 41 controls the timing at which at least one power LED module 22 emits light via the power LED driver 21 and distributes signals with time as shown in FIG. The opening / closing timings of the transfer gates TX1 and TX2 of each light receiving unit of the array unit 311 are controlled reciprocally.
The host interface unit 42 is an interface for exchanging necessary information by communicating with a system application processor 5 such as a playground equipment 11 to be controlled, which will be described later with reference to FIG. This is an interface for information communication of human motion information to the application processor 5 in three-dimensional space coordinates.

  In the data format unit 43, the pixel array unit 311 includes, for example, 80,000 pixels as a plurality of light receiving units, and the sensor output from the TOF distance image sensor 31 is output in parallel in units of two pixels in order to increase the processing speed. Since the output information per pixel is output in the form of ΔV1 and ΔV2 of the output voltages of the two terminals, the data format is converted to facilitate the arithmetic processing as the subsequent signal processing. Yes.

  In addition, the data format unit 43 includes a distance frame generation unit 431 that generates a distance frame (Range) in a predetermined format, and a distance balance frame generation unit 432 that generates a distance balance frame (background image frame) in a predetermined format. And have.

  The distance frame generation unit 431 generates a distance frame by calculating the distance L from [Expression 2] based on the output voltage information of the TOF distance image sensor 31 of [Expression 1].

  The distance balanced frame generation unit 432 generates a distance balanced frame that serves as a reference (background image frame) for frame difference calculation when detecting a human motion based on at least two distance frames. The distance balanced frame generation unit 432 will be described in more detail below.

  The distance displacement detection unit 44 calculates the distance L of the projection space of the projection light based on the distance information from the TOF type distance image sensor 31 (the output voltage information corresponding to the distance and the information of Expression 1 above) [Expression 2]. Based on the measured distance L (distance frame), the frame difference calculation unit 46 calculates the amount of change in the distance of the projection space by frame difference calculation processing (RFsubRF) between the distance frames. The presence or absence of a distance displacement can be detected by comparing with a threshold value.

  That is, the distance displacement detection unit 44 determines whether or not there is a change in distance between distance frames in order to determine whether or not a person has performed an action. For example, distance information for 120 frames per second enters the distance displacement detection unit 44 from the distance image sensor 31. In the distance displacement detection unit 44, the amount of change with respect to time of the distance L is determined by the frame difference calculation unit 46. It is obtained from (RFsubRF). Here, the frame difference between at least two distance frames is calculated by the frame difference calculator 46 here, but the distance displacement detector 44 may calculate a frame difference between at least two distance frames. .

  Each distance frame has a temporal fluctuation of the distance due to a temporal fluctuation (random noise) of the output voltage information of the TOF type distance image sensor 31. Therefore, the person does not operate at all and the change of the distance L changes. Even if not, the frame difference (RFsubRF) does not become “0” in all pixels. Therefore, a threshold (frame difference threshold) for the change in the distance L is set in order to determine whether or not a person has moved. Considering the distance resolution performance of the TOF distance image sensor 31, a plurality of threshold values for the change in the distance L are prepared in a table format for each representative distance L that is greater than or equal to the distance resolution of the TOF distance image sensor 31. Keep it.

  When the frame difference (RFsubRF) is less than the set threshold, the distance displacement detection unit 44 determines that there is no change in the distance of the projection light in the projection space (the target object A is not moving). On the other hand, when the frame difference (RFsubRF) is equal to or greater than the set threshold value, the distance displacement detection unit 44 determines that there is a change in the distance of the projection light in the projection space (the target object A is moving).

  The distance equilibrium state determination unit 45 determines whether the projection space of the projection light is in a distance equilibrium state. The distance equilibrium state determination unit 45 determines that the projection space of the projection light is a distance if the period in which the distance displacement detection unit 44 determines that there is no time change of the distance L continues for a predetermined time or more than a predetermined number of frames. Judged to be in equilibrium. Here, the frame difference between at least two distance frames is calculated by the frame difference calculation unit 46 here, but the distance equilibrium state determination unit 45 may calculate the frame difference between at least two distance frames. Good.

  The frame difference calculation unit 46 performs frame difference calculation (RFsubRF) between continuously acquired distance frames, or performs frame difference calculation (RFsubBRF) between the distance frame and the distance balanced frame.

  As described above, the frame difference calculation unit 46 calculates the frame difference between each of the plurality of distance frames and the distance balanced frame (background image), and from the frame difference, for example, the changed horizontal direction and vertical direction (XY coordinates). An area having the largest area, an area having the largest change amount of the distance L, and the like are calculated as the movement information of the person, and the change amount and the change direction of the distance are calculated.

  The distance balanced frame generation unit 432 stores at least one continuous distance frame determined by the distance displacement detection unit 44 that there is no change in distance from among the continuously acquired distance frames. When the distance equilibrium state determination unit 45 determines that the projection space of the projection light is in the distance equilibrium state, the distance equilibrium frame generation unit 432 calculates the time average of at least two stored distance frames, and the distance equilibrium A frame (Balanced Range Frame; BRF) is generated. By taking this time average, the temporal random noise included in the distance balanced frame is reduced, and erroneous detection when detecting the human motion by taking the frame difference between the distance frame and the distance balanced frame can be reduced.

  The distance balanced frame generation process does not need to be frequently performed at a predetermined time (for example, once every second). For example, the distance balanced frame generation process is performed when the high-speed human motion detection device 1 is activated or when the application processor 5 initializes a game program. Just do it. Therefore, since the calculation amount of the distance balanced frame generation process is large but the frequency is low, real-time processing at the time of driving at a high frame rate is not hindered.

(Embodiment 2)
In the first embodiment, the human motion detection device 1 according to the present invention that obtains the distance L to the target object A three-dimensionally has been described. However, in the second embodiment, the high-speed human motion detection device 1 is the target. A case will be described in detail where the present invention is applied to a playground equipment that controls the game content and progress according to the motion content of the person as the object A.

  FIG. 5 is a block diagram illustrating a configuration example of a main part of the playground equipment according to the second embodiment of the present invention. In addition, the same code | symbol is attached | subjected and demonstrated to the structural member which show | plays the effect similar to the structural member of FIG.

  In FIG. 5, the playground equipment 11 of the second embodiment includes various controls necessary for the progress of the game, such as execution of a game program and exchange of distance information with the human motion detection device 1 of the first embodiment. And an application processor 5 for performing

  Since the human motion detection device 1 includes the TOF type distance image sensor 31, it can greatly reduce the amount of calculation in the subsequent signal processing, and in particular, a playground equipment that requires a high frame rate such as a video game. 11 is suitable for real-time processing.

  The application processor 5 advances the game by controlling the content of the game, such as reflecting the movement of the person in the movement of the character in the game or the occurrence of an event based on the three-dimensional space coordinates of the movement information of the person.

  The human motion detection unit 51 of the application processor 5 compares the frame differences (RFsubRF and RFsubBRF) acquired continuously in order to acquire detailed information on the motion of the person to be detected. Various information such as the position of the region (distance and direction), the speed vector of the person's motion (speed and direction of motion), and the acceleration vector of the motion of the person (magnitude and direction of motion acceleration) are extracted. The position of the motion area is obtained as a solid coordinate (XYZ coordinate) from the position in the horizontal / vertical direction (XY coordinate) of the distance frame and the distance L (distance L from the human motion detection device 1 orthogonal to the XY plane). The speed vector and acceleration vector of a person's movement may be obtained from the average movement distance of the movement area, or may be obtained from the movement distance of the center of gravity position of the movement area. For a specific position (for example, a person's head, arm, or hand) You may ask for it.

  Here, between the human motion detection device 1 and the human motion detection unit 51, the host is configured to limit the signal processing only to the distance information required by the application processor 5 in accordance with the content and progress of the game. Information may be exchanged via the interface unit 42. That is, on the game program side of the application processor 5, the time at which the person who is the target object A expects to perform an action (hereinafter, the expected time) or the presence of the person is determined according to the content and progress of the game. It has information on an expected space (hereinafter, an expected existence space) and a space where a person is expected to operate (hereinafter, an expected motion change).

  FIG. 6 is a diagram schematically showing an expected time and an expected existence space in the best mode for carrying out the second embodiment, and FIG. 7 is an expectation in the best mode for carrying out the second embodiment. It is a figure which shows typically time, expected presence space, and expected operation | movement change.

  6 and 7, the expected time, the expected presence space, and the expected motion change will be described in detail by taking a tennis game as an example. The expected time is when the person who is the target object A is expected to take some action. For example, at time t0 in FIG. 6, since tennis has not started in the game, the person who is the target object A is not expected to move or move. At the next time t1, the person who is the target object A is just moving in the order in which the opponent in the game returns, and here the person and the opponent who are the target object A still move. And is not expected to work. Furthermore, at the next time t2, it is the turn in which the players in the game return, and the person who is the target object A is expected to start moving to the position where it reaches the ball.

  An expected presence space is a space area in which a person as the target object A is expected to move. For example, at time t2 in FIG. 6, when a player in the game has to move to the right of the tennis court in order to return, the person who is the target object A is expected to move right to the expected existence space P2. Yes. Further, the expected motion change is the speed and direction of motion expected to be performed by the person who is the target object A.

  At time t3 in FIG. 7, the person who is the target object A moves to the right to the expected existence space P3. When the ball comes near the player in the game at time t4 in FIG. 7, the person who is the target object A is It is expected that a return ball motion (expected motion change V4) is performed with a racket held in the hand in the expected presence space P4.

  In the application processor 5, since only the expected motion change V in the expected existence space P at the expected time t is meaningful as the game program progresses, it can be considered that other monitoring amounts are unnecessary. Therefore, the human motion detection unit 51 only needs to perform a calculation process for detecting a human motion only for the distance information that matches each expected value.

  Since the human motion detection unit 51 can reduce the amount of calculation by not performing unnecessary signal processing, real-time processing can be performed even when the human motion detection device 1 is driven at a high frame rate. In addition, the possibility of erroneous detection of an operation irrelevant to the game progress of the person who is the target object A or an operation other than the person who is the target object A when a plurality of persons are in the imaging region is reduced.

  Here, first, a simple and particularly high-speed human motion detection process of the human motion detection device 1 of FIG. 5 will be described with reference to a flowchart shown in FIG.

  As shown in FIG. 8, first, in step S1, the high-speed person motion detection control means 4 uses the light emitting means 2 or the light receiving means 3 in response to an instruction to turn on the power or an operation start from the game program of the playground equipment 11 or the like. For example, distance frames are acquired at a frame rate of about 120 frames per second.

  Next, a frame difference is calculated in step S2. That is, the frame difference calculation unit 46 of the high-speed human motion detection control means 4 performs frame calculation between distance frames. Since the magnitudes of the output terminal voltages ΔV1 and ΔV2 are different by one to two digits when compared with a region where the person who is the target object A is present and a region where the person is not present (background), the presence and movement of the person can be detected. Is possible.

  Thereafter, the calculation result information is passed to the application processor 5 via the host interface unit 42. In step S3, the human motion detection unit 51 of the application processor 5 determines whether or not the distance displacement matches the expected value of the application processor 5. If they match, in step S4, “there is a human motion”. If it does not match, it is determined in step S5 that there was no person action.

  The simple and high-speed human motion detection control step includes a distance frame generation step in which the distance frame generation unit 431 generates a plurality of distance frames in time series based on distance information from the TOF type distance image sensor 31, and a frame difference. A calculation unit 46 has a frame difference calculation step of calculating each frame difference for a plurality of distance frames, and calculating a change amount and a change direction of the distance of the person as the motion information of the person from the calculated frame difference. Yes.

  Next, a highly accurate human motion detection process of the high-speed human motion detection device 1 of FIG. 1 will be described with reference to the flowchart shown in FIG.

  As shown in FIG. 9, first, in step S11, the high-speed human motion detection control unit 4 uses the light emitting unit 2 or the light receiving unit 3 in accordance with an instruction to turn on the power or start the operation from the game program of the playground equipment 11. For example, distance frames are acquired at a frame rate of about 120 frames per second.

  Next, in step S12, the high-speed human motion detection control unit 4 starts an operation for obtaining a frame difference for each distance frame at a predetermined timing.

  In step S13, it is determined whether or not the frame difference (RFsubRF) is less than a set threshold value. When the frame difference (RFsubRF) is less than the set threshold value, the distance displacement detection unit 44 determines that “there is no distance displacement” in step S14. If the frame difference (RFsubRF) is greater than or equal to the set threshold value, the distance displacement detection unit 44 determines that “distance displacement is present” in step S15, and then returns to the distance frame acquisition process in step S11.

  Furthermore, when the state determined as “no distance displacement” by the distance displacement detection unit 44 in step S14 continues for a predetermined time or more or a predetermined number of frames, the distance equilibrium state determination unit 45 projects the projection light. It is determined that the space is in a distance equilibrium state.

  Thereafter, in step S16, when the distance balanced state determination unit 45 determines that the projection space of the projection light is in the distance balanced state, the distance balanced frame generation unit 432 takes a time average of at least one distance frame to obtain the distance balanced frame. Is generated.

  Subsequently, an image showing a specific example of the distance balanced frame is shown in FIG. In FIG. 10, there is no person who is the target object A and the distance balanced frame is generated only from the background. However, even if the person is in the imaging range, for example, the person is moved to the right side by an instruction from the game program of the playground equipment 11 The distance frame of the left half is generated, and then the person is moved to the left side to generate the distance frame of the right half, and finally the distance balanced frame is generated by combining them. Alternatively, if the person is stationary, the distance displacement can be detected by frame difference calculation (RFsubBRF) even if the distance balanced frame including the stationary person is generated.

  The above highly accurate human motion detection control step includes a distance frame generation step in which the distance frame generation unit 431 generates a plurality of distance frames in time series based on the distance information from the TOF type distance image sensor 31, and a distance balanced frame. The generation unit 432 generates a distance balanced frame that is distance information that does not change the distance of the person based on at least two distance frames, and the frame difference calculation unit 46 sets the distance for each of the plurality of distance frames. There is a frame difference calculation step for calculating a frame difference from the balanced frame and calculating a change amount and a change direction of the distance of the person as the motion information of the person from the calculated frame difference.

  FIGS. 11 and 12 respectively show images showing specific examples of distance information acquired by the distance frame generation unit 431 in FIG. 5 at a certain time and subsequent times.

  In step S <b> 17, the frame difference calculation unit 46 performs a frame difference calculation between the distance frame and the distance balanced frame. The frame difference (RFsubBRF) in FIGS. 10 and 11 and the frame difference (RFsubBRF) in FIGS. 10 and 12 are shown in FIGS. 13 and 14, respectively. Areas without distance displacement are canceled out by frame difference calculation, and only areas with distance displacement remain.

  FIG. 15 shows the frame difference between FIG. 13 and FIG. It is detected how much the person who is the target object A has moved while acquiring the distance frames shown in FIGS. 11 and 12, respectively.

  Thereafter, the calculation result information is passed to the application processor 5 via the host interface unit 42. In step S18, the person motion detection unit 51 determines whether or not the distance displacement matches the expected value of the application processor 5. If they match, it is determined in step S19 that "there is a person motion". If they do not match, it is determined in step S20 that “no human motion has occurred”.

(Embodiment 3)
In the first embodiment, the human motion detection device 1 according to the present invention that three-dimensionally calculates the distance L to the target object A will be described, and in the second embodiment, the high-speed human motion detection device 1 is the target object. A case where the present invention is applied to a playground equipment that controls game content and progress according to the motion content of a person as A will be described. In the third embodiment, the frame difference calculation (RFsubRF) in step S17 of FIG. A method for determining whether or not the whole body of the person who is the target object A has moved by adding the subsequent steps will be described in detail.

  FIG. 22 is a block diagram illustrating a configuration example of a main part of the playground equipment according to the third embodiment of the present invention. In addition, the same code | symbol is attached | subjected and demonstrated to the structural member which show | plays the effect similar to the structural member of FIG. 1 and FIG.

  In FIG. 22, the playground equipment 11A of the third embodiment includes distance information between the person motion detection device 1A that three-dimensionally calculates the distance L to the target object A at a plurality of points, and the execution of the game program and the person motion detection device 1A. And an application processor 5A for performing various controls necessary for the progress of the game, such as exchange of information.

  The human motion detection device 1A further includes a center coordinate detection unit 47 in the configuration of the human motion detection device 1 of the first and second embodiments.

  A distance balanced frame generating unit 432 as a distance balanced frame generating unit generates a distance balanced frame (BRF), and a frame difference calculating unit 46 as a frame difference calculating unit calculates a difference between the Nth distance frame and the distance balanced frame ( RFsubBRF) is calculated. Thereby, the person who is the target object A can be cut out and detected from the background information having no distance change. That is, the frame difference calculation unit 46 calculates a frame difference between the distance frame and the distance balanced frame, and detects a person based on whether or not there is a change in distance from the background information in which the distance information is in a balanced state.

  In this state, the center coordinate detection unit 47 as the center coordinate detection means obtains the center coordinates and the movement amount of the center coordinates from the distance information of the person detected by the frame difference between the distance frame and the distance balanced frame. That is, the center coordinate detection unit 47 obtains center coordinates from the detected distance information of the Nth distance frame. For the next N + 1th distance frame, the center coordinates are obtained in the same manner, and the movement amount of the center coordinates between the Nth and N + 1th distance frames is obtained. In this case, the center coordinates may be obtained, for example, by calculating the center-of-gravity coordinates from the distance information of the whole body of the detected person, and the midpoint from the maximum value and the minimum value of the X, Y, Z coordinates of the whole body of the detected person. You may obtain | require a coordinate and may obtain | require the coordinate of the point from which the sum of the distance from each point of the detected person's outline becomes the minimum.

  That is, the center coordinate detection unit 47 calculates the center-of-gravity coordinates of each coordinate axis in the three-dimensional space as the center coordinates from the detected distance information of the whole body of the person. Alternatively, the center coordinate detection unit 47 calculates the midpoint as the center coordinate from the maximum value and the minimum value in each coordinate axis direction of the three-dimensional space from the detected distance information of the whole body of the person. Alternatively, the center coordinate detection unit 47 calculates, from the detected distance information of the entire body of the person, a point at which the sum of the distances from the points on the contour of the person is the minimum as the center coordinates.

  The application processor 5A advances the game by controlling the content of the game, such as reflecting the action of the person in the action of the character in the game or the occurrence of an event based on the three-dimensional space coordinates of the action information of the person.

  The human motion detection unit 51A as the human motion detection unit of the application processor 5A compares the movement amount of the central coordinate between distance frames acquired successively with the expected value in the game, thereby determining the central coordinate movement determination unit. The center coordinate movement determination unit 511 determines that the person has “whole body movement” when the movement amount of the person's center coordinates is equal to or greater than a threshold value set based on the expected value. If the movement amount is less than the threshold, it is determined that the person has not moved the whole body.

  Here, only the distance information required by the application processor 5A is signaled between the high-speed human motion detection control means 4A and the human motion detector 51 of the human motion detector 1A according to the game content and progress. Information may be exchanged via the host interface unit 42 so as to limit processing. That is, on the game program side of the application processor 5A, the time at which the person who is the target object A expects to perform an action (hereinafter, the expected time) or the presence of the person is determined according to the content and progress of the game. It has information on an expected space (hereinafter, an expected existence space) and a space where a person is expected to operate (hereinafter, an expected motion change).

  Next, with reference to the flowchart shown in FIG. 23, the highly accurate human motion detection process of the high-speed human motion detector 1A of FIG. 22 will be described.

  As shown in FIG. 23, first, in step S11, the high-speed human motion detection control unit 4A uses the light emitting unit 2 or the light receiving unit 3 in accordance with an instruction to turn on the power or start an operation from the game program of the playground equipment 11A. For example, distance frames are acquired at a frame rate of about 120 frames per second.

  Next, in step S12, the high-speed human motion detection control unit 4A starts an operation for obtaining a frame difference for each distance frame at a predetermined timing.

  In step S13, it is determined whether or not the frame difference (RFsubRF) is less than a set threshold value. When the frame difference (RFsubRF) is less than the set threshold value, the distance displacement detection unit 44 determines that “there is no distance displacement” in step S14. If the frame difference (RFsubRF) is greater than or equal to the set threshold value, the distance displacement detection unit 44 determines that “distance displacement is present” in step S15, and then returns to the distance frame acquisition process in step S11.

  Furthermore, when the state determined as “no distance displacement” by the distance displacement detection unit 44 in step S14 continues for a predetermined time or more or a predetermined number of frames, the distance equilibrium state determination unit 45 projects the projection light. It is determined that the space is in a distance equilibrium state.

  Thereafter, in step S16, when the distance balanced state determination unit 45 determines that the projection space of the projection light is in the distance balanced state, the distance balanced frame generation unit 432 takes a time average of at least one distance frame to obtain the distance balanced frame. Is generated.

  Further, in step S17, the frame difference calculation unit 46 performs a frame difference calculation between the distance frame and the distance balanced frame. The area without the distance displacement is canceled by the frame difference calculation, and only the area with the distance displacement remains.

  Thus, according to the distance balanced frame generation method, the distance balanced frame (BRF) is obtained (step S16), and the difference (RFsubBRF) between the Nth distance frame and the distance balanced frame is calculated (step S17). The person who is the target object A can be cut out and detected from the background information in which the distance is in an equilibrium state, that is, the distance does not change.

The steps up to this point are the same as the steps up to the frame difference calculation (RFsubRF) in step S17 in FIG. 9, but whether the whole body of the person who is the target object A has moved by adding steps after step S17. Whether it can be determined.
That is, in step S21 following the frame difference calculation process, the center coordinate detection unit 47 of the high-speed person motion detection control unit 4A obtains center coordinates from the detected person distance information for the Nth distance frame. For example, the center coordinates may be obtained by determining the center-of-gravity coordinates from the distance information of the whole body of the detected person, and the center point coordinates are obtained from the maximum and minimum values of the X, Y, Z coordinates of the whole body of the detected person. Alternatively, the coordinates of the point that minimizes the sum of the distances from the points of the detected contour of the person may be obtained. Similarly, the center coordinates are similarly obtained for the next N + 1th distance frame, and the movement amount of the center coordinates between the Nth and N + 1th distance frames is obtained. For example, in a tennis game, when a person moves to the ball return position, if the person moves back and forth and right and left in the projection space of the projection light, the center coordinates of the person move between distance frames, so where the person's body moves from where It is possible to obtain an index value that is simple in calculation and has a small amount of calculation.

  Further, in step S22, the information on the movement amount of the center coordinate is passed to the application processor 5A via the host interface unit. On the game program side, there is an expected value where and how much the person should move according to the game content, and the person motion detection unit 51A of the application processor 5A compares the movement amount of the center coordinates of the person with the expected value.

  If the movement amount of the central coordinate of the person is equal to or greater than the threshold value set from the expected value, in the next step S23, the central coordinate movement determination unit 511 not only moves the limb of the person who is the target object A. It is determined that “the whole body has moved”. On the other hand, if the movement amount of the central coordinate of the person is less than the threshold value, the central coordinate movement determination unit 511 determines that “the whole body has not moved” in step S24. In this way, it is possible to determine whether or not the whole body of the person who is the target object A has moved.

(Embodiment 4)
In the third embodiment, the method for determining whether or not the whole body of the person who is the target object A has moved has been described. However, in the fourth embodiment, the central coordinate calculation process of the third embodiment is used in the detection of the human motion. A method for determining which part of the body the distance displacement between the distance frames corresponds to by adding the subsequent steps will be described.

  FIG. 24 is a block diagram illustrating a configuration example of a main part of a playground equipment according to Embodiment 4 of the present invention. In addition, the same code | symbol is attached | subjected and demonstrated to the structural member which show | plays the effect similar to the structural member of FIG. 1 and FIG.

  In FIG. 24, the playground equipment 11B of the fourth embodiment includes distance information between the person motion detection device 1B that three-dimensionally calculates the distance L to the target object A from a plurality of points, and the execution of the game program and the person motion detection device 1A. And an application processor 5B that performs various controls necessary for the progress of the game, such as exchange of information.

  The human motion detection device 1B further includes a central coordinate detection unit 47B in the configuration of the human motion detection device 1 of the first and second embodiments.

  The center coordinate detection unit 47B obtains each center coordinate from the detected person distance information for the Nth distance frame and the N + 1th distance frame, and calculates the movement amount of the center coordinates between the Nth and N + 1th distance frames. In addition, the relative position between the distance displacement and the center coordinate is calculated from the distance displacement obtained by the difference calculation between the distance frames and the center coordinate obtained by the frame difference calculation between the distance frame and the distance balanced frame.

  The application processor 5B advances the game by controlling the game content based on the three-dimensional spatial coordinates of the movement information of the person, such as reflecting the movement of the person in the movement of the characters in the game and the occurrence of an event.

  The human motion detection unit 51B of the application processor 5B compares the amount of movement of the center coordinates between the distance frames acquired successively with the expected value in the game, and determines the center coordinate movement as the center coordinate movement determination means. The unit 511 determines that the person has “whole body movement” when the movement amount of the central coordinates of the person is equal to or greater than the threshold set from the expected value, and the movement amount of the central coordinates of the person is less than the threshold If it is, it is determined that the person has not moved.

  In addition, the human motion detection unit 51B of the application processor 5B has a body part determination unit 512 as a body part determination unit that determines the body part of the person who has operated based on the relative positional relationship between the center coordinate and the distance displacement of the distance frame difference. have. The body part determination unit 512 compares the expected value with at least one of the time when the body part exists as a result of the determination, the spatial region, and the action change, and the time, the spatial region, and the action change when the body part exists. If at least one of the values matches the expected value, it is determined that the body part has “expected motion”, and at least one of the time, space region, and motion change at which the body part exists is expected. If the values do not match, the body part is determined to have “no expected motion”. In short, the body part determination unit 512 determines which part of the body of the person that is the target object A has moved from the relative positional relationship between the center coordinates and the coordinates of the distance displacement.

  On the game program side, there is an expected value as to what body part the person should move according to the game content, and compared with the center coordinates located at the center of the body, for example, the distance between distance frames If the displacement is in the upper right (horizontal and vertical XY coordinate directions) from the center coordinates, it is the right arm movement, the upper left is the left arm movement, the lower right is the right leg movement, and the lower left is the left leg movement. Is determined. If the determination result of the body part determination unit 512 matches the expected value, it is determined that “there is an expected action”. If the determination result of the body part determination unit 512 does not match the expected value, the “expected action” is determined. It was determined that there was no.

  Here, only the distance information required by the application processor 5B is signaled between the high-speed human motion detection control unit 4B and the human motion detector 51B of the human motion detector 1B according to the content and progress of the game. Information may be exchanged via the host interface unit 42 so as to limit processing. That is, on the game program side of the application processor 5B, the time at which the person who is the target object A expects to perform an action (hereinafter referred to as the expected time) and the presence of the person according to the content and progress of the game. It has information on an expected space (hereinafter, an expected existence space) and a space where a person is expected to operate (hereinafter, an expected motion change).

Next, with reference to the flowchart shown in FIG. 25, the highly accurate human motion detection process of the high-speed human motion detector 1B of FIG. 24 will be described. Here, steps having the same effects as the steps of the flowchart shown in FIG. 23 are denoted by the same reference numerals, and description thereof is omitted.
In the detection of the human motion, it is possible to determine which part of the body the distance displacement between the distance frames corresponds to by adding the steps after the central coordinate calculation process of the third embodiment.

That is, in step S31, the distance displacement obtained by the difference calculation (RFsubRF) between the distance frames in step S12 and the central coordinates obtained by the frame difference calculation (RFsubBRF) between the distance frame and the distance balanced frame in step S21. Calculate the relative position of the distance displacement and center coordinates.
In the same manner as the movement amount of the center coordinate between the distance frames obtained in step S21 is passed to the application processor 5B via the host interface unit 42, the distance displacement obtained in step S12 and the relative position of the center coordinate are determined in step S31. The relative distance between the obtained distance displacement and center coordinates is passed to the application processor 5B via the host interface unit 42. On the game program side, there is an expected value of how much the person should move in which direction according to the game content.
In the application processor 5B, first, in step 22, as to whether or not the entire body of the person has moved, the movement amount of the center coordinates is compared with the expected value as in the case of the third embodiment.

  Further, as in the case of the third embodiment, when the movement amount of the center coordinates is equal to or larger than the threshold set from the expected value, the center coordinate movement determination unit 511 simply determines that the person who is the target object A is the target object A in step S23. In addition to moving the limbs, it is determined that the person has “whole body movement”. On the other hand, if the movement amount of the center coordinates is less than the threshold value, in step S24, the center coordinate movement determination unit 511 determines that “the whole body has not moved”. Thereby, it can be determined whether or not the whole body of the person who is the target object A has moved.

  Next, in step 32, the body part determination unit 512 determines which part of the body of the person that is the target object A has moved, based on the relative positional relationship between the center coordinates and the distance displacement coordinates. Can do. That is, the game program side has an expected value as to what body part the person should move according to the game content, and compared with the center coordinates located at the center of the body, for example, between the distance frames If the distance displacement is right upper than the center coordinates (horizontal and vertical XY coordinate directions), the right arm moves, the upper left moves the left arm, the lower right moves the right foot, and the lower left moves the left foot. It is determined that

When the determination result of the body part determination unit 512 matches the expected value, it is determined in step 33 that “there is an expected motion”. On the other hand, if the determination result of the body part determination unit 512 does not match the expected value, it is determined in step 34 that “the expected motion has not occurred”. In this way, it is possible to determine to which part of the body the distance displacement between the distance frames corresponds.
As described above, the method according to the fourth embodiment for determining which part of the body the distance displacement between the distance frames corresponds to is obtained by extracting the human skeleton from the feature points such as the conventional joints. Compared to the modeling method, the calculation amount is significantly less, and it is suitable for a play equipment such as a video game machine having a high required frame rate.

(Embodiment 5)
In the second embodiment, the case where the human motion detection device 1 is used for the playground equipment 11 that controls the content and progress of the game according to the motion content of the person of the target object A will be described. The method for determining whether or not the entire body of the person who is the object A has moved has been described. In the fourth embodiment, the method for determining which part of the body the distance displacement between the distance frames corresponds to has been described. In the fifth embodiment, a case will be described in which the playground equipment according to the second to fourth embodiments further includes a sensation generating device for generating a sense of reality by generating an impact or impact sound.

  FIG. 26 is a block diagram illustrating a configuration example of a main part of the playground equipment according to the fifth embodiment of the present invention. In addition, the same code | symbol is attached | subjected and demonstrated to the structural member which show | plays the effect similar to the structural member of FIG. 1 and FIG. FIG. 26 shows a case where the playground equipment 11B of the fourth embodiment further includes a sensation generating device for generating a sense of reality by generating an impact or impact sound. FIG. 27 shows a case where the playground equipment device 11 further includes a bodily sensation stimulation generating device, and FIG. 28 shows a case where the playground equipment device 11A of the third embodiment further includes a bodily sensation stimulation generating device. Here, an example of a sensation generating device will be described with reference to FIG.

  In FIG. 26, when the application processor 5C of the playground equipment 11C determines that there is a human motion that matches the expected value in the progress of the game, the human motion detector 51C detects the motion, and the wireless transmitter 57 (for example, Bluetooth) The sensory stimulation control signal is transmitted to the sensory stimulation generating device 8 through a wireless communication using a wireless communication or a wired connection.

  The bodily sensation generating device 8 includes a wireless receiving unit 81 that receives the bodily sensation stimulation control signal from the wireless transmission unit 57, and a vibration generating unit 82 (for example, a vibrator) that generates an impact such as a ball hitting a tennis racket in the case of a tennis game. A light emitting unit 83 (for example, LED) that emits light, an audio output unit 84 (for example, a speaker) that generates an impact sound such as a ball hitting a tennis racket in the case of a tennis game, and a scent generating unit 85 (for example, that generates a scent) A door that encloses an object that emits a scent), and in response to the received signal, the housing of the sensation generator 8 generates vibrations or shocks, emits light, or sounds such as a collision sound. To generate a scent. Thereby, the person (person who is the target object A) holding the bodily sensation stimulus generating device 8 can obtain a sense of reality by obtaining feedback according to the game contents and the operation contents of the person. In short, the bodily sensation stimulus generating device 8 generates at least one of vibration, light, sound, and scent.

  Here, a tennis game in which the sensation stimulus generator 8 is regarded as a racket will be described as an example.

  When the racket which is the sensation generating device 8 hits the ball, the center coordinate detection unit 47B obtains the center coordinate from the detected distance information of the person and further obtains the movement amount of the center coordinate between the distance frames. The relative position between the distance displacement and the center coordinates is calculated from the distance displacement obtained by the difference calculation of the distance and the center coordinates obtained by the frame difference calculation between the distance frame and the distance balance frame. Next, the human motion detection unit 51B of the application processor 5C compares the distance displacement of the distance frame difference acquired successively, the relative position of the center coordinates, and the expected value in the game, and the body part determination unit 512 Indicates that, based on the relative positional relationship between the center coordinates and the distance displacement coordinates, the target object A has a racket in the body of the person, here the right hand. Determine.

  On the game program side, there are expected values for the body part, right hand, and racket that the person should act on according to the game content, and if the determination result of the body part determination unit 512 matches the expected value, If the determination result of the body part determination unit 512 does not match the expected value, it is determined that there was no expected action. In this case, the expected value is an expected value related to at least one of the time when the body part is present in the human motion, the spatial region, and the motion change.

  When the body part determination unit 512 determines that “there is an expected motion”, that is, at the moment when the racket hits the ball, the application processor 5C performs a human motion (a motion to swing the racket) that matches the expected value in the progress of the game. ), The human motion detection unit 51B detects the motion, and transmits a bodily sensation stimulus control signal from the application processor 5C to the bodily sensation stimulus generation device 8 through the wireless transmission unit 57.

  In the bodily sensation generating device 8, when the bodily sensation stimulation control signal from the wireless transmission unit 57 is received by the wireless receiving unit 81, the impact of the ball hitting the housing of the bodily sensation generating device 8 using the internal vibration generating unit 82. The voice output unit 84 is used to generate a collision sound that hits the ball.

  Thereby, the person holding the sensation generating device 8 which is a racket can feel the impact at the moment when the ball hits the racket by hand. Further, since the sound is not heard from the television speaker but from the sensation stimulating device 8 held in the hand, it is possible to obtain an operational feeling and a sense of reality that are closer to real tennis. It should be noted that the vibration, voice strength, length, rhythm, and the like may be changed according to the speed of the ball, the direction in which the ball flies, the swing speed of the racket, the presence or absence of a shift in the timing of the collision, and the like.

  The bodily sensation stimulus generation device 8 according to the fifth embodiment corresponds to a remote controller in a playground equipment 11C such as a conventional video game machine. However, it is not limited to a function that wirelessly transmits a user's operation as a control signal to the playground equipment 11C, but may be dedicated to a function that receives a control signal from the playground equipment 11C.

  When a human motion is detected by an acceleration sensor or an angular velocity sensor mounted inside a remote controller as in a conventional playground equipment device, depending on the game content, a vibration generator such as a vibrator mounted inside the remote controller as described above. The remote controller could not be vibrated. This is because the acceleration sensor and the angular velocity sensor cannot distinguish between the human motion and the vibration of the remote controller, and therefore, when it is vibrated, only the intended human motion cannot be detected. In the fifth embodiment, since the human motion is detected only by the human motion detector 1, 1 </ b> A, or 1 </ b> B outside the bodily sensation stimulus generator 8, the above problem does not occur even if the bodily sensation stimulus generator 8 is vibrated. In bodily sensation-type games such as sports, feedback by tactile sensation to the user is as important as visual and auditory senses, and can greatly improve the user's operational feeling.

  In the above description of the tennis game as an example, the case where the user holds the sensation stimulus generating device 8 as a racket in the hand has been described. If it exists, it may be held in any part of the body, and it may be installed in a range that can be sensed by the user if it is light, voice, or scent. The sensory stimulation generator 8 generates at least one of vibration, light, voice, and scent. In the tennis game, “vibration” is used instead of impact, and “sound” is used instead of the impact sound when the ball is hit. For other “light” cases, for example, the LED can be lit, for example, a part or all of the sword can be made to shine instead of the spark when the “swords” collide with each other. As an example of “scent”, various “scents” can be generated by heating a medicine with a heater, and for example, it can be used in place of the smell of a shell after shooting a ball with a pistol.

  As described above, according to the first to fifth embodiments, the signal charges distributed depending on the flight time (delay time) of the light corresponding to the distance L to the target object A are two pixel output electrodes V1 and V2, respectively. The signal charges are converted into output voltage information corresponding to the distance L and output from the TOF distance image sensor 31 as distance information. Based on the distance information from the TOF type distance image sensor 31, the light flight time (delay time) T is detected with the known speed of light as c, and the target object is obtained from L = (1/2) · c · T. The distance L to A is obtained, and based on this, the distance displacement detecting means 44 for detecting the distance displacement of the projection light to the target object A in the projection space and the distance for determining whether or not the projection space is in the distance equilibrium state The balance determination means 45 is included, and it is detected whether or not the person who is the target object A has operated based on the frame difference (RFsubBRF). As a result, when a large number of spatial regions are processed in a plane, the amount of calculation is not enormous, unlike the conventional distance image sensor, even at a high frame rate. In addition to performing in real time, it is possible to detect with high accuracy the detection accuracy of the direction of human motion and the amount of change in distance in the imaging range.

  By detecting this highly accurate and computationally inexpensive human action, the game contents and progress of the playground equipment 11 and 11A-11E such as a video game can be controlled in real time, and the playground equipment 11 and 11A-11E. The operability can be improved.

  Further, the person motion detection device 1, 1A or 1B is used without using the game controller used in the conventional playground equipment, and the three-dimensional human motion is detected using the person motion detection device 1, 1A or 1B. A game is performed without using a game controller by driving a sensory stimulus generator that generates at least one of vibration, light, voice, and scent by performing a comparison process between the detection result of the person motion and the expected value. It became possible to do. This eliminates the expense of creating a game controller, and further eliminates the need to press the game controller with your finger. We can expect improvement of technique in sports.

  In the second to fifth embodiments, the person motion detection device 1, 1A or 1B of the present invention is used for the playground equipment 11 and 11A to 11E for controlling the game content and progress according to the person motion. However, the present invention is not limited to this, and any electronic device can be used as long as it can be controlled using the human motion detection device 1, 1A, or 1B that detects human motion information in a three-dimensional space in real time. But you can.

  Although not particularly described in Embodiments 2 to 5, the application processors 5 and 5A to 5E are provided with various components in addition to the human motion detectors 51, 51A, and 51B. This will be described in detail below. Here, the application processor 5 of the second embodiment will be described as a representative.

  FIG. 16 is a block diagram illustrating a hardware configuration example of a main part of the application processor 5 of FIG.

  In FIG. 16, the application processor 5 according to the second embodiment acquires detailed information on the CPU (central processing unit) 50 as a control unit that performs overall control and the action of the person to be detected as described above. Therefore, by comparing the frame differences (RFsubRF and RFsubBRF) acquired continuously, the position (distance and direction) of the person's motion region, the speed vector (speed and direction of motion) of the person, The person motion detection unit 51 that extracts various information such as the motion acceleration vector (the magnitude and direction of motion acceleration), and various controls necessary for the progress of the game using the person motion detection information by the person motion detection unit 51 A game execution unit 52 and a keyboard, mouse, touch panel, and pen input device for inputting commands to the CPU 50, and further communication An operation unit 53 such as an input device that receives and inputs via a network (for example, the Internet or an intranet), and a display unit that displays an initial screen, a selection scene, a control result screen based on a game program, an operation input screen, and the like on the display screen 54, ROM 55 as a computer-readable readable recording medium storing the game program and its data, and the game program and its data are read out at the time of activation, and the data is read and stored for each control by CPU 1 It has RAM56 as a memory | storage part which works as a work memory.

  The CPU 50 reads out a game program and its data as a control program stored in the ROM 55 into the RAM 56 when the apparatus is activated, and the person motion detection unit 51 is continuously acquired based on the game program and its data. Frame differences (RFsubRF and RFsubBRF), the position (distance and direction) of the person's motion area, the speed vector (motion speed and direction) of the person's motion, and the acceleration vector (motion acceleration of the motion of the person) A person action detecting step for extracting various information such as size and direction), and a game executing step in which the game executing unit 52 performs various controls necessary for the game progress using the person action detecting information by the person action detecting unit 51. It is supposed to run.

  The ROM 55 as a readable recording medium may be composed of a hard disk, a formable optical disk, a magneto-optical disk, a magnetic disk, an IC memory, and the like. Although this game program and its data are stored in the ROM 55, this game program and its data may be downloaded to the ROM 55 from another readable recording medium or via wireless, wired or the Internet.

  Similarly, in the first to fifth embodiments, the CPU of the high-speed human motion detection control means 4 (not shown) reads the human motion detection program as a control program stored in the ROM and its data into the RAM when the apparatus is activated. Based on the person motion detection program and the data, the timing control unit 41 controls the drive timing of the power LED driver 21 and the TOF distance image sensor 31, respectively, and the host interface unit 42 A host interface step for communicating with a system such as the playground equipment 11 on the host side, and the data format unit 43 outputs an output data stream based on distance information (output voltage information corresponding to the distance) from the TOF distance image sensor 31. Convert to the specified format A data format step, a distance displacement detection unit 44 that detects a distance displacement in the projection space of the projection light, and a distance equilibrium state determination unit 45 that determines whether or not there is a distance displacement. The frame difference calculation unit 46 executes a frame difference calculation step as a frame difference calculation unit for obtaining a frame difference between the distance frame and the distance balanced frame, and the person as the target object A based on the light reception information from the light receiving unit 3 The operation information of is detected.

  In this data formatting step, the human motion detection program and its data as a control program stored in the ROM are read into the RAM when the apparatus is activated, and the distance frame generation unit 431 is based on the human motion detection program and its data. The distance frame generation step for generating a plurality of distance frames in time series based on the distance information from the TOF type distance image sensor 31 and the distance balanced frame generation unit 432 generate a person based on at least two distance frames. A distance balanced frame generation step of generating a distance balanced frame (background image frame) which is distance information without a distance change is executed.

  Also in this case, as described above, the ROM as the readable recording medium may be composed of a hard disk, a formable optical disk, a magneto-optical disk, a magnetic disk, an IC memory, and the like. This person motion detection program and its data are stored in the ROM, but this person motion detection program and its data are downloaded to the ROM from another readable recording medium or via wireless, wired or the Internet. May be.

  Although not particularly described in the first to fifth embodiments, the light emitting means 2 that emits the projection light and the reflected light from the projection space of the projection light are received and the distance to the person in the projection space is determined. A person motion detection device 1 having a TOF distance image sensor 31 that outputs the distance information from a plurality of light receiving units, and detects person motion information based on the distance information from the TOF distance image sensor 31. High-speed human motion detection control means 4 is provided. The high-speed human motion detection control means 4 includes a distance frame generation unit 431 that generates a plurality of distance frames in time series based on distance information from the TOF type distance image sensor 31, and a person based on at least two distance frames. A distance balanced frame generation unit 432 that generates a distance balanced frame (background image frame) that is distance information that does not change in distance, and calculates a frame difference from the distance balanced frame for each of the plurality of distance frames. A frame difference calculation unit 46 that calculates a distance change amount and a change direction of a person as motion information.

  As a result, when a large number of spatial regions are subjected to arithmetic processing, the amount of calculation does not become enormous even at a high frame rate, and the detection accuracy of the amount of change in the direction and distance of human motion in the imaging range can be detected with high accuracy. The object of the present invention can be achieved.

  As mentioned above, although this invention has been illustrated using preferable Embodiment 1-5 of this invention, this invention should not be limited and limited to this Embodiment 1-5. It is understood that the scope of the present invention should be construed only by the claims. It is understood that those skilled in the art can implement an equivalent range based on the description of the present invention and the common general technical knowledge from the description of specific preferred embodiments 1 to 5 of the present invention. Patents, patent applications, and documents cited herein should be incorporated by reference in their entirety, as if the contents themselves were specifically described herein. Understood.

  The present invention relates to a human motion detection device and a human motion detection method for detecting a human motion, a playground equipment device that detects a human motion using the human motion detection device, and performs various controls of game content. The distance from the TOF type distance image sensor in the field of the game method used, the control method for causing the computer to execute the person motion detection method and the game method, and the computer readable recording medium storing the control program Based on the information (output voltage information corresponding to the distance), means for generating a plurality of distance frames in time series, and based on at least one of the distance frames, a distance balanced frame that is a distance frame having no distance change is provided. Calculate the frame difference between the means to generate and the distance balanced frame for each distance frame, and the distance of the target object Even if it has a means to calculate the amount of change and the direction, and it detects a change in the distance and direction to the area where the distance displacement occurred as a person's motion, even when processing a large number of spatial areas in a plane Thus, unlike the conventional distance image sensor and triangulation distance image sensor using the intensity of reflected light, the amount of calculation is not enormous, and human motion detection at a high frame rate can be performed in real time. By detecting this person action, various control of game contents can be performed in real time according to the person action, and the operability of the playground equipment such as a video game can be greatly improved.

DESCRIPTION OF SYMBOLS 1, 1A, 1B Human motion detection apparatus 2 Light emission means 21 Power LED driver 22 Power LED module 22a Power LED
DESCRIPTION OF SYMBOLS 23 Radiation angle adjustment lens 3 Light receiving means 31 TOF type distance image sensor 311 Pixel array part 312 Timing generation part 313 Vertical scanner part 314 Horizontal scanner part 315 Pixel driver part 316 Noise canceller part 317 A / D converter part 32 Viewing angle adjustment lens 4, 4A, 4B High-speed human motion detection control means 41 Timing control unit 42 Host interface unit 43 Data format unit 431 Distance frame generation unit 432 Distance equilibrium frame generation unit 44 Distance displacement detection unit 45 Distance equilibrium state determination unit 46 Frame difference calculation unit 47 47B Central coordinate detection unit 5, 5A to 5E Application processor 51, 51A to 51E Human motion detection unit 511 Central coordinate movement determination unit 512 Body part determination unit 52 Game execution unit 53 Operation unit 54 Display unit 5 ROM
56 RAM
57 wireless transmission unit 8 bodily sensation generating device 81 wireless receiving unit 82 vibration generating unit 83 light emitting unit 84 audio output unit 85 scent generating unit 6 schematic diagram of game contents 7 television 11, 11A to 11E play equipment A target object T light of Flight time c Speed of light C1 Capacitance at the pixel output electrode V1 C2 Capacitance at the pixel output electrode V2 PG Charge conversion area TX1, TX2 Charge transfer gate V1, V2 Pixel output electrode ΔV1, ΔV2 Output voltage at the pixel output electrode T0 Pulse width Td Reflected light delay time Iph Photocurrent due to reflected light

Claims (48)

A light emitting means for emitting the projection light, and a TOF (Time-Timer) that receives the reflected light from the projection space of the projection light and outputs distance information according to the distance to the person in the projection space from each of the plurality of light receiving units. A human motion detection device having a distance image sensor,
A human motion detection device having human motion detection control means for detecting human motion information based on distance information from the TOF distance image sensor. The person motion detection control means includes
Distance frame generating means for generating a plurality of distance frames in time series based on distance information from the TOF type distance image sensor;
2. The frame difference calculating unit according to claim 1, further comprising: a frame difference calculating unit that calculates a frame difference for each of the plurality of distance frames, and calculates a change amount and a change direction of the distance of the person as motion information of the person from the frame difference. Human motion detection device. The person motion detection control means includes
Distance frame generating means for generating a plurality of distance frames in time series based on distance information from the TOF type distance image sensor;
Distance balanced frame generating means for generating a distance balanced frame that is distance information without a change in distance to the person based on at least two of the distance frames;
Frame difference calculating means for calculating a frame difference with the distance balanced frame for each of the plurality of distance frames, and calculating a change amount and a change direction of the distance of the person as motion information of the person from the frame difference. Item 2. The person motion detection device according to Item 1.   The distance displacement detection means according to claim 2 or 3, further comprising distance displacement detection means for measuring the distance to the person in real time and detecting the presence or absence of a change in the distance to the person from a frame difference between at least two distance frames. Human motion detection device.   When the distance between the person is measured in real time and the frame difference between at least two distance frames is less than a predetermined threshold value continues for a predetermined time or more or a predetermined number of frames, the distance information of the person The person motion detection device according to claim 3, further comprising a distance equilibrium state determination unit that determines that is in an equilibrium state.   6. The distance balanced frame generating unit according to claim 5, wherein the distance balanced frame generating unit generates a distance balanced frame by calculating a time average of at least two distance frames when the distance balanced state determining unit determines that the distance balanced state is in a distance balanced state. Human motion detection device.   The distance balanced frame generation means moves the person to the right or left side of the imaging range in accordance with an instruction from the playground equipment when the person is within the imaging range, and the imaging of the left or right half of the imaging range A distance-balanced frame of one half that does not have a person in the range is generated, the person is moved to the left or right side of the imaging range, and the other half that does not have the person in the imaging range on the right or left half of the imaging range 4. The human motion detection device according to claim 3, wherein a distance balanced frame is generated by synthesizing a distance balanced frame having no person in the imaging range. 6. The human motion detection according to claim 5, further comprising a frame difference calculation means for calculating a frame difference between the distance frame and the distance balanced frame, and detecting a person based on the presence or absence of a change in distance from a background in which the distance information is in a balanced state. apparatus.   The person motion detection device according to claim 8, further comprising center coordinate detection means for obtaining center coordinates of the person from distance information of the person detected by a frame difference between the distance frame and the distance balanced frame.   The human motion detection device according to claim 9, wherein the central coordinate detection unit obtains a movement amount of the central coordinate between the distance frames.   The center coordinate detection means is configured to calculate a relative position between the distance displacement and the center coordinate from a distance displacement obtained by a difference calculation between the distance frames and a center coordinate obtained by a frame difference between the distance frame and the distance balanced frame. The human motion detection device according to claim 10, wherein   The human motion detection device according to claim 9, wherein the central coordinate detection unit calculates, as the central coordinate, a barycentric coordinate of each coordinate axis in a three-dimensional space from the detected distance information of the entire human body.   The human motion detection according to claim 9, wherein the central coordinate detection means calculates a midpoint as the central coordinate from the maximum value and the minimum value in each coordinate axis direction of the three-dimensional space from the detected distance information of the entire human body. apparatus.   The human motion according to claim 9, wherein the central coordinate detection unit calculates, as the central coordinate, a point having a minimum sum of distances from each point on the contour of the person from distance information of the detected person's entire body. Detection device.   The human motion detection device according to claim 3, wherein the distance balanced frame generation unit performs the process of generating the distance balanced frame at the time of starting the apparatus, at initialization, or at predetermined time intervals.   The human motion detection device according to claim 1, wherein the light emitting unit projects near-infrared pulsed light as the projection light onto the projection space at a constant period.   The light receiving unit is provided with a photoelectric conversion unit in the center, and two pixel output electrodes V1 via two charge transfer gates to which opposite gate signals are input at a point symmetrical position with respect to the center of the photoelectric conversion unit. , V2 are provided to output signal charges distributed depending on the time of flight of light corresponding to the distance to the person from the two pixel output electrodes V1, V2, respectively. Detection device.   18. The signal charges from the two pixel output electrodes V1 and V2 are converted into output voltage information corresponding to the distance to the person, respectively, and output as the distance information from the TOF type distance image sensor. The human motion detection device according to 1.   18. The distance L to the person is obtained from L = (1/2) · c · T by detecting a flight time T of light to the person with a known speed of light as c. The human motion detection device described.   The person motion detection device according to claim 1, wherein the distance information is three-dimensional distance information for each pixel corresponding to the light receiving unit.   A motion or event of a character in the game using the human motion detection device according to any one of claims 1 to 20 and the motion information of the distance and direction of the distance from the human motion detection device. And an application processor that controls the game to reflect the occurrence of the game.   The application processor compares the frame difference information acquired continuously in order to acquire the detailed information of the motion of the person, the position information of the motion area of the person, the velocity vector of the motion of the person, and the motion The playground equipment according to claim 21, further comprising a person motion detection unit that extracts various pieces of information on the acceleration vector.   The position information of the motion area is obtained three-dimensionally from the horizontal and vertical XY coordinate positions and distances of the distance frame, and the velocity vector and acceleration vector of the motion of the person are the average moving distance of the motion area, the The playground equipment according to claim 22, wherein the playground equipment is obtained from one of a movement distance of the center of gravity position of the motion region and a movement distance of the specific part position of the person.   The person has an expected value regarding at least one of the time of the body part of the action performed by the person, the space area, and the action change, and at least one of the time, the space area, and the action change that matches the expected value The playground equipment according to claim 22, wherein the processing by the person motion detection means is performed only for a distance frame including the distance information.   When the movement amount of the central coordinate of the person is greater than or equal to a threshold set from the expected value, it is determined that the person has moved the whole body, and the movement amount of the central coordinate of the person is the threshold The playground equipment according to claim 22 or 24, further comprising central coordinate movement determination means for determining that "the whole body has not moved" when the number is less than.   The playground equipment according to claim 22 or 24, further comprising body part determination means for determining a body part of an operated person from a relative positional relationship of a distance displacement of a difference between the center coordinate and the distance frame.   The body part determination means, if the distance displacement detected by the frame difference between the distance frames is upper right from the center coordinate, the right arm movement, if the distance displacement is upper left from the center coordinate, the left arm movement, the right from the center coordinate 27. The playground equipment according to claim 26, wherein it is determined that the motion is a right foot if it is below, and a left foot motion if it is below the center coordinates.   The body part determination means compares the expected value with at least one of the time, the spatial region, and the movement change when the body part exists as the determination result, and when the at least one matches the expected value 28. The method according to claim 26 or 27, wherein it is determined that the body part has “expected motion”, and if at least one of the body parts does not match the expected value, the body part is determined to have “no expected motion”. A playground equipment according to claim 1.   As the expected value, the timing when the person is expected to take some action is assumed as the expected time, the space area where the person is expected to perform the moving operation is assumed as the expected existence space, and the person will perform The playground equipment according to any one of claims 24 to 27, wherein an expected motion speed or direction is an expected motion change.   The application processor has an expected value related to at least one of the time of the person's action, the spatial region, and the action change, and when detecting a person action that matches the expected value, the application processor The playground equipment according to claim 21, further comprising a bodily sensation generating device that generates at least one of vibration, light, voice, and fragrance.   A person motion is detected using the person motion detection device without using a game controller mounted on a conventional playground equipment device, a comparison process between the detection result of the person motion and the expected value is performed, vibration, The playground equipment according to claim 21, wherein the device for generating a bodily sensation stimulus that generates at least one of light, sound, and scent is driven. The reflected light from the projection space of the projection light emitted from the light emitting means is received, and distance information according to the distance to the person in the projection space is received by a plurality of light receptions of a TOF (Time of Flight) distance image sensor. A human motion detection method for outputting from each of the sections,
A human motion detection method comprising a human motion detection control step of detecting human motion information based on distance information from the TOF distance image sensor. The person motion detection control step includes:
A distance frame generating step for generating a plurality of distance frames in time series based on distance information from the TOF type distance image sensor;
The frame difference calculating step according to claim 32, further comprising: calculating a frame difference for each of the plurality of distance frames, and calculating a change amount and a change direction of the distance of the person as motion information of the person from the frame difference. Human motion detection method. The person motion detection control step includes:
A distance frame generating step for generating a plurality of distance frames in time series based on distance information from the TOF type distance image sensor;
A distance balanced frame generating step of generating a distance balanced frame that is distance information without a distance change for the person based on at least two of the distance frames;
A frame difference calculating step of calculating a frame difference from the distance balanced frame for each of the plurality of distance frames, and calculating a change amount and a change direction of the person's distance as motion information of the person from the frame difference. Item 33. The person motion detection method according to Item 32.   The person motion according to claim 33 or 34, further comprising a distance displacement detecting step of measuring the distance of the person in real time and detecting the presence or absence of a change in the distance of the person from a frame difference between at least two distance frames. Detection method.   When the distance of the person is measured in real time and the state where the frame difference between at least two distance frames is less than a predetermined threshold continues for a predetermined time or more or a predetermined number of frames, the distance information of the person is The person motion detection method according to claim 34, further comprising a distance equilibrium state determination step for determining that the state is in an equilibrium state. 37. A human motion operation according to claim 36, further comprising a frame difference calculation step of calculating a frame difference between the distance frame and the distance balanced frame, and detecting the person based on presence or absence of a change in distance from a background whose distance information is in a balanced state. Detection method.   38. The person motion detection method according to claim 37, further comprising a center coordinate detection step of obtaining center coordinates from distance information of the person detected by a frame difference between the distance frame and the distance balanced frame.   The person motion detection method according to claim 38, wherein the center coordinate detection step obtains a movement amount of the center coordinates between the distance frames.   The center coordinate detection step includes calculating a relative position between the distance displacement and the center coordinate from the distance displacement obtained by the difference calculation between the distance frames and the center coordinate obtained by the frame difference between the distance frame and the distance balanced frame. 40. The human motion detection device according to claim 38 or 39, wherein   39. The person motion detection method according to claim 38, wherein the center coordinate detection step calculates a center-of-gravity coordinate of each coordinate axis in the three-dimensional space as the center coordinate from the detected distance information of the entire body of the person.   The center coordinate detection step calculates a center point as the center coordinate from the maximum value and the minimum value of each coordinate axis direction in the three-dimensional space from the detected distance information of the whole body of the person. The human motion detection method described.   39. The human motion according to claim 38, wherein the central coordinate detection step calculates, as the central coordinate, a point having a minimum sum of distances from each point on the contour of the person from distance information of the detected whole body of the person. Detection method.   The application processor reflects the movement of the character in the game and the occurrence of the event using the movement amount of the person's distance from the person movement detection device according to any one of claims 1 to 20 and the movement information of the changing direction. The game method which has a game execution step which controls a game so that it may be made.   In order to acquire the detailed motion information of the person, the frame difference information acquired continuously is compared, and the position information of the motion area of the person, various information on the speed vector and acceleration vector of the motion of the person are extracted. A person motion detecting step, and using information on the amount of change in the person's distance and the direction of the person's distance, as well as position information on the person's motion region, various information on the speed vector and acceleration vector of the person's motion 45. The game method according to claim 44, wherein, as the game execution step, the game is controlled so that the application processor reflects the action of a character in the game or the occurrence of an event.   44. A control program in which a processing procedure for causing a computer to execute the steps of the human motion detection method according to claim 32 is described.   46. A control program in which a processing procedure for causing a computer to execute the steps of the game method according to claim 44 or 45 is described.   A computer-readable readable recording medium in which the control program according to claim 46 or 47 is stored.