JP2018142815A - Three-dimensional data acquisition device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To continue acquisition of three-dimensional data in a state where the ratio of an object occupying the field angle of a three-dimensional sensor is high.SOLUTION: An acquisition unit 11 acquires three-dimensional data indicating the three-dimensional position of each point on the object detected by a 3D sensor 31, a recognition unit 12 recognizes the position and the posture of the object based on the three-dimensional data, an estimation unit 13 estimates the posture of the object at a time Tn+1, based on a received activity list 21 and the posture of the object recognized by the recognition unit 12, with reference to an operation pattern DB23 indicating multiple related operations including the same posture, and storing multiple operation patterns defined by transition of the state indicating the posture of the object, and a control unit 17 controls the orientation and the field angle of the 3D sensor 31 based on the estimated posture and position of the object.SELECTED DRAWING: Figure 1

Description

  The disclosed technology relates to a three-dimensional data acquisition apparatus and three-dimensional data acquisition.

  Conventionally, there is a technique for tracking an object using a three-dimensional sensor capable of acquiring three-dimensional data indicating the three-dimensional position coordinates of the object.

  For example, there has been proposed a moving object detection apparatus that simultaneously realizes moving object detection in a wide area and output of a moving object image with high resolution. This apparatus includes a first image capturing unit that captures a wide-angle image and a second image capturing unit that captures a narrow-angle image having a narrower angle of view than the wide-angle image. Further, this apparatus detects the position of a moving object in a wide-angle image, cuts out a moving object image including the moving object from the wide-angle image based on the position of the moving object, creates feature image information indicating the characteristics of the moving object from the moving object image, The feature image information and the narrow-angle image are collated. In addition, the apparatus predicts the moving position of the moving object in the wide-angle image after a predetermined time based on the detected position of the moving object, and the second imaging unit detects the position of the moving object based on the predicted position information of the moving object. The position or direction of the second imaging unit is moved so as to capture the moving object in the field of view.

JP 2005-72840 A

  In the prior art, when the moving object moves on the extension line of the movement history, it is possible to predict and track the moving destination of the moving object. However, in the case of an object whose movement direction changes suddenly, such as a gymnast, the conventional technique cannot appropriately predict the movement destination of the object at the next timing. Tracking accuracy is reduced.

  In addition, when it is desired to grasp the detailed operation of the object, it is desirable that the acquisition of the three-dimensional data can be continued in a state where the ratio of the object in the angle of view of the three-dimensional sensor is high.

  An object of the disclosed technique is to continue acquiring three-dimensional data in a state in which a ratio of an object occupying a field angle of a three-dimensional sensor is high.

  As one aspect, the three-dimensional data acquisition device includes an acquisition unit that acquires three-dimensional data indicating the three-dimensional position of each point on the object detected by the three-dimensional sensor, and the 3D acquired by the acquisition unit. A recognition unit for recognizing the position and orientation of the object based on the dimension data. In addition, the three-dimensional data acquisition apparatus refers to an operation pattern storage unit that shows a plurality of related operations including the same posture and stores a plurality of operation patterns defined by state transitions indicating the posture of the object. And an estimation unit. The estimation unit estimates the posture of the target object at the next timing based on the received information on the operation scheduled to be performed by the target object and the posture of the target object recognized by the recognition unit. Further, the estimation unit estimates a position where the object is present at the next timing based on the estimated posture and a change in the position of the object recognized by the recognition unit. In addition, the three-dimensional data acquisition apparatus includes a control unit that controls the orientation and the angle of view of the three-dimensional sensor based on the posture and position of the object estimated by the estimation unit.

  As one aspect, acquisition of three-dimensional data can be continued in a state where the ratio of the object in the angle of view of the three-dimensional sensor is high.

It is a functional block diagram of the three-dimensional data acquisition device according to the first and second embodiments. It is a figure which shows an example of a performance schedule list. It is a figure which shows an example of a history database (DB). It is a figure which shows an example of operation | movement pattern DB. It is a figure for demonstrating the local coordinate system centering on the advancing direction of operation | movement. It is a figure for demonstrating a sensor coordinate system. It is a figure for demonstrating calculation of the transition probability according to the difference of the attitude | position which a present attitude | position and a state candidate show. It is a figure for demonstrating calculation of direction and angle of view of 3D sensor. It is a block diagram which shows schematic structure of the computer which functions as a three-dimensional data acquisition apparatus which concerns on 1st and 2nd embodiment. It is a flowchart which shows an example of the three-dimensional data acquisition process in 1st and 2nd embodiment. It is a flowchart which shows an example of a state determination process. It is a flowchart which shows an example of a state estimation process. It is a figure which shows an example of a connection operation | movement table. It is a figure for demonstrating normalization and correction | amendment of the size of a human body model.

  Hereinafter, an example of an embodiment according to the disclosed technology will be described in detail with reference to the drawings. In each of the following embodiments, a case will be described in which an object to be tracked is a gymnast performing performance.

<First Embodiment>
As shown in FIG. 1, the three-dimensional data acquisition apparatus 10 according to the first embodiment is connected to a three-dimensional sensor (hereinafter referred to as “3D sensor”) 31.

  The 3D sensor 31 detects, for example, the distance and direction to the object with a predetermined resolution by a TOF (Time Of Flight) method, and outputs three-dimensional data indicating the three-dimensional position coordinates of each point on the object. . The 3D sensor 31 can change the angle of view indicating the detection range in the horizontal direction and the vertical direction, and the direction of the center line connecting the center of the detection range and the 3D sensor 31, and the angle of view indicated by the set control value and It drives by the drive part 32 so that it may become direction.

  The 3D data acquisition device 10 uses the 3D sensor 31 so that the 3D sensor 31 can continue to acquire 3D data in a state in which the proportion of the object in the 3D sensor 31 is high. Control the orientation.

  Here, the reason for acquiring the three-dimensional data in a state where the ratio of the object in the angle of view of the 3D sensor 31 is high will be described.

  For example, a TOF 3D sensor receives a reflection of laser light applied to an object, and measures a distance to each point on the object from an elapsed time from irradiation to reflection of the laser light. When the distance between the object and the 3D sensor is 1 m, the time required for the laser beam to reciprocate is 6.7 ns. Further, since the irradiation light is attenuated by the square of the distance to the object, it is necessary to receive very weak laser light when receiving light.

  In order to achieve both the distance to the object and the high-definition of the obtained three-dimensional data, the 2D scan type TOF is advantageous from the viewpoint of improving the intensity of irradiation light, but in the case of the 2D scan type, the object Maximum theoretical distance x (6.7 ns + discrimination margin) x number of distance measuring points in one frame x frame rate is 1 second or less. If the maximum distance to the object is 10 m and the acquisition is performed at 30 fps, the number of distance measurement points for one frame is less than 500,000 points (about 800 × 600 pixels) even if the time margin for determination is ignored. Become. That is, compared with a normal visible camera, it is difficult to photograph at a high speed and a high resolution with a 3D sensor.

  For example, consider a case where a 3D sensor having a resolution of 320 × 240 (QVGA) acquires three-dimensional data with a gymnast as an object. If the detection range covering the whole body of a player who is not moving is about 3 m × 3 m, the measurement can be performed with less than about 1 cm between one pixel. However, assuming that the player is moving, if the detection range is about 10 m × 6 m, the distance between one pixel is about 2 to 3 cm. In this case, although depending on the orientation of the player's body with respect to the 3D sensor, for example, a portion where the 3D data cannot be detected, such as the tip of the player's limb, can be generated, and the body line and the straightness of the limb can be analyzed in detail There is a possibility that 3D data cannot be acquired.

  Therefore, even with a 3D sensor having a relatively low resolution, by acquiring 3D data so that the entire body of the object is contained within the angle of view of the 3D sensor, it is possible to acquire 3D data of the details of the object. To be possible, it is necessary to track the object with a 3D sensor.

  As illustrated in FIG. 1, the three-dimensional data acquisition apparatus 10 includes an acquisition unit 11, a recognition unit 12, a state determination unit 14, a state estimation unit 15, and a position estimation unit 16, an estimation unit 13, and a control unit 17. Including. In addition, in a predetermined storage area of the three-dimensional data acquisition apparatus 10, a performance schedule list 21, a history database (DB) 22, and an operation pattern DB 23 are stored.

  The acting schedule list 21 is a list in which techniques scheduled to be performed by a gymnast as an object are described in time series. FIG. 2 shows an example of the performance schedule list 21. FIG. 2 shows an example in which technique numbers, which are identification information of techniques scheduled to be included from the start (Start) to the end (end) of an action, are described in the order of the execution. The acting schedule list 21 is input to the three-dimensional data acquisition device 10 and stored in a predetermined storage area before the target gymnast starts acting.

  The history DB 22 stores the human body model recognized based on the three-dimensional data acquired from the 3D sensor 31, the determined state, and the history of the technique performed. FIG. 3 shows an example of the history DB 22. In the example of FIG. 3, the history DB 22 includes a human body model history table 22A, a state history table 22B, and a technique execution history table 22C.

  In the human body model history table 22A, a coordinate value group of each part of the human body constituting the human body model recognized by the recognition unit 12 is stored in time series. Each part that constitutes the human body model is a part of the human body such as a joint that is a feature point for specifying the posture of the object. For example, each part is the center of the head, shoulder, right shoulder, right elbow, right wrist, right hand, left shoulder, left elbow, left wrist, left hand, spine, buttocks, right hip, right knee, right ankle, right foot, These include the left hip, left knee, left ankle, and left foot. As the coordinate value of each part, a vector indicating the axial direction of each part and a quaternion representing rotation can be used. In the example of FIG. 3, an example in which the coordinate value of each part is stored in association with the index of each part is shown.

  In the state history table 22B, the state (details will be described later) of the object determined by the state determination unit 14 is stored in time series. In the example of FIG. 3, the state history table 22B stores a state number that is identification information of the determined state in association with the time.

  In the technique execution history table 22C, technique numbers of techniques determined to be completed by the state determination unit 14 are stored in time series. In the example of FIG. 3, the data structure of the technique performance history table 22C is the same as the data structure of the performance schedule list 21 shown in FIG.

  The motion pattern DB 23 stores a plurality of motion patterns defined by state transitions indicating a plurality of related motions including the same posture and indicating the posture of the object. Specifically, the motion pattern is an operation indicating the standard technique and other techniques that can be changed from the posture of the target object during the execution of the standard technique as a plurality of related operations including the same posture. And the operation when the execution of the standard technique and other techniques is not established or fails.

  FIG. 4 shows an example of the operation pattern DB 23. As shown in FIG. 4, in the present embodiment, each motion pattern is divided into a node corresponding to a state indicating each posture included in the motion pattern, and an edge connecting between nodes corresponding to transitionable states. It is expressed as a tree structure that contains it. In FIG. 4, nodes in which numbers are written are nodes corresponding to states indicating postures in the middle of techniques, and numbers are state numbers corresponding to the respective nodes. A shaded node is a node corresponding to a state indicating an attitude of a technique (for example, an upright attitude, an attitude with a hand raised, an attitude hanging on a horizontal bar or a hanging ring). The double circle node is a node corresponding to a state indicating a posture at the time of failure such as a fall or fall.

  In the example of FIG. 4, the operation pattern will be described in more detail by taking as an example a tree structure indicating an operation pattern having an operation pattern number of 1 which is identification information of the operation pattern. In the following, the motion pattern with the motion pattern number i is “pattern i”, the state with the state number j is “state j”, the skill with the skill number k is “skill k”, and the state j included in the pattern i Is expressed as “pattern i: state j”.

  The first stage part of pattern 1 starts from the posture indicating the break of the technique, and shows the posture of the break of the technique through the intermediate postures indicated by the states 1, 2,. , It represents a change in posture when technique 1 is established. Also, during the implementation of Technique 1, the player may intentionally change to a different technique, or the player may unintentionally lose balance, resulting in a technique lower than Technique 1. , Subordinate techniques may not be established or may end in failure.

  A route that branches from state 5 to technique 2 and a route that branches from state 4 to technique 3 represent a change in posture in the case of a lower technique. In addition, as an example of becoming a low-order technique, there is a case where a three-time twist becomes a two-time twist or a one-time twist. In addition, the route that branches from the state 3 and passes through the state 11, the state 12, and the state 13 to the failure of the technique is not rotated, dropped, etc. This represents a change in posture when the technique is not established. Further, a path that branches from the state 11 and reaches the failure through the state 14 represents a change in posture when the execution of the technique 1, technique 2, or technique 3 fails.

  The motion pattern DB 23 also stores state information defining postures indicated by the states included in the motion pattern. The state information includes a reference coordinate system, a coordinate value group of each part of the human body in the reference coordinate system, and information on an allowable error range. The coordinate value group of each part of the human body is the same as the coordinate value group of each part constituting the human body model stored in the above-described human body model history table 22A. In addition, the coordinate value group of each part included in the state information of each state stored in the motion pattern DB 23 is a coordinate value group of each part constituting a human body model of a reference size assuming a standard human body size. is there.

  As a reference coordinate system, for example, a relative coordinate system based on the coordinate value of a predetermined part of the human body, a global coordinate system set on the basis of an instrument such as a horse or a floor performance area, and the direction of progress of the performance You can define a local coordinate system with axes. The local coordinate system with the acting direction as an axis is a coordinate system used for state information about a state indicating a posture of a technique that does not depend on global coordinates, such as a technique performed on the floor.

  The allowable error range is information indicating the allowable error range from the reference posture indicated by the coordinate value group of each part of the human body. For example, as shown in FIG. 4, the allowable error range is defined in a relative coordinate system such as an angle range formed by both feet, or is defined in a global coordinate system such as an angle range from an axis set based on an instrument. You can do it. Also, for example, as shown in FIG. 5, a local coordinate system with the direction of the foot at the time of crossing as an axis is set for the human body, and the range of the angle between the direction of the foot at the time of crossing and the direction of the foot at the time of landing You may define

  Next, each functional unit of the three-dimensional data acquisition apparatus 10 will be described.

  The acquisition unit 11 acquires three-dimensional data indicating the three-dimensional position of each point on the object detected by the 3D sensor 31. As shown in FIG. 6, the three-dimensional data is data in which a three-dimensional coordinate value of each point included in a point group on the object detected by the 3D sensor 31 at each time Tn is expressed in a sensor coordinate system. The three-dimensional coordinate value of the point P at time Tn is defined as PTn (Xsn, Ysn, Zsn). The acquisition unit 11 passes the acquired three-dimensional data group to the recognition unit 12.

  Further, the acquisition unit 11 acquires information on the direction and angle of view of the 3D sensor 31 at time Tn together with the three-dimensional data, and passes the information to the estimation unit 13. The direction of the 3D sensor 31 at time Tn is expressed as θTn (θpan_n, θtilt_n, θroll_n) using the pan angle θpan, the tilt angle θtilt, and the roll angle θroll. The angle of view of the 3D sensor 31 at time Tn is expressed as φTn (φxs_n, φys_n) using the horizontal field angle φxs and the vertical field angle φys in the sensor coordinate system. When the position of the 3D sensor 31 in the global coordinate system changes, the acquisition unit 11 also acquires position information in the global coordinate system of the 3D sensor 31 at time Tn from the 3D sensor 31 and sends it to the estimation unit 13. Deliver.

  The acquisition unit 11 acquires the input performance schedule list 21 and stores it in a predetermined storage area.

  The recognition unit 12 recognizes a human body model indicating the position and orientation of the object in the sensor coordinate system based on the three-dimensional data group transferred from the acquisition unit 11. Specifically, the recognition unit 12 calculates coordinate values (for example, quaternion expression vector values) for each part of the human body described above from the three-dimensional data group. Since the conventional technique can be applied to the method of calculating the coordinate value of each part of the human body from the three-dimensional data group, detailed description thereof is omitted here. The recognizing unit 12 normalizes the calculated coordinate values of each part to the size of the human model serving as a reference in consideration of the physique difference for each player as the object.

  The recognizing unit 12 stores the human body model represented by the calculated coordinate value group of each part in the human body model history table 22A of the history DB 22 in association with the acquisition time Tn of the three-dimensional data and the index of each part. . For example, using this index (body index), the coordinate value in the sensor coordinate system of each part constituting the human body model at time Tn can be expressed as body index_Tn (xs, ys, zs, ws).

  The estimation unit 13 estimates the position and orientation of the object at the next timing (time Tn + 1). As described above, the estimation unit 13 includes the state determination unit 14, the state estimation unit 15, and the position estimation unit 16.

  The state determination unit 14 refers to the performance schedule list 21 and the motion pattern DB 23 to determine which state the posture of the target human body model recognized by the recognition unit 12 includes in which motion pattern.

  Specifically, the state determination unit 14 receives the coordinate value group of the sensor coordinate system at the time Tn stored in the human body model history table 22A of the history DB 22 and the orientation and image of the 3D sensor 31 transferred from the acquisition unit 11. Using the corner information, it is converted into a coordinate value group in the global coordinate system. The state determination unit 14 updates the coordinate value at time Tn stored in the human body model history table 22A to the coordinate value converted into the global coordinate system.

  In addition, the state determination unit 14 refers to the state history table 22B of the history DB 22, and when the state at the time Tn-1 indicates a posture of a technique break, the technique execution history of the performance schedule list 21 and the history DB 22 Referring to the table 22C, the next technique after the completed technique is specified. Then, the state determination unit 14 acquires an operation pattern including the specified technique from the operation pattern DB 23, and in each of the acquired operation patterns, the state indicating the first posture (the state next to the state indicating the posture of the technique delimiter) ) Is extracted as a state candidate at time Tn. For example, when the next technique is technique 1, pattern 1 is acquired from operation pattern DB 23 shown in FIG. 4, and pattern 1: state 1 is extracted as a state candidate at time Tn. If there are other motion patterns including technique 1 in the motion pattern DB 23, state candidates are also extracted from the motion patterns.

  In addition, when the state at the time Tn−1 is a state indicating a posture in the middle of the technique, the state determination unit 14 is a state that can transition from the state at the time Tn−1 in the operation pattern including the state at the time Tn−1. Are extracted as state candidates at time Tn. For example, when the state at time Tn−1 is state 3 of pattern 1 shown in FIG. 4, pattern 1: state 3, pattern 1: state 4, and pattern 1: state 11 are extracted as state candidates at time Tn. The The state candidate “pattern 1: state 3” is a case where the state has not changed since time Tn−1.

  The state determination unit 14 compares the state information of each extracted state candidate with the human body model stored in the human body model history table 22A of the history DB 22, and determines the state at time Tn. For example, the state determination unit 14 calculates the Euclidean distance between the coordinate value group included in the state information and the coordinate value group included in the human body model between corresponding portions, and the sum of the Euclidean distances of the respective portions is the smallest. The state candidate at the time Tn can be determined. In the present embodiment, since the human body model stored in the human body model history table 22A is normalized to the reference size, comparison with the state information of each state stored in the operation pattern DB 23 is facilitated.

  The state determination unit 14 stores the state number of the determined state together with the pattern number of the operation pattern including the state in association with the time Tn in the state history table 22B of the history DB 22.

  The state estimation unit 15 extracts a state that can transition from the state at the time Tn determined by the state determination unit 14 as a state candidate at the time Tn + 1. The method for extracting the state candidates is the same as the case where the state determination unit 14 extracts the state candidates at time tn.

  The state estimation unit 15 calculates a transition probability from the state at time Tn for each of the extracted state candidates. For example, a transition probability corresponding to the level of difficulty of the technique performed through the transition between the states is given in advance between the states of each operation pattern. For example, in the pattern 1 of FIG. 4, when the difficulty level of the technique 1 is higher than the technique 2, the transition probability from the state 5 to the state 8 is higher than the transition from the state 5 to the state 6. A transition probability is given between the states so as to increase. The difficulty level of a technique can use the difficulty level and value point defined by the scoring standard.

  Moreover, you may make it change a transition probability according to a player's level. For example, when the technique to be implemented is technique 1, the higher the level, the higher the transition probability between the states included in the route to technique 1, and the lower the level, the lower level technique and technique. The transition probability between the states included in the path leading to failure or failure can be increased. The level of the player can be determined from, for example, the world ranking or the score in the most recent game. In this way, the state estimation unit 15 can use the transition probability given in advance between the states as the transition probability from the state at the time Tn to each state candidate.

  In addition, for each state included in the motion pattern, a standard time for maintaining the posture indicated by the state is defined, and the state estimation unit 15 is an object with respect to the standard time with reference to the state history table 22B. Find the ratio of actual run time of the players. Then, the state estimating unit 15 may calculate a transition probability when the current state is maintained and a transition probability when transitioning to the next state according to the obtained ratio.

  In addition, as shown in FIG. 7, the state estimation unit 15 makes a transition according to the difference between the posture indicated by each state candidate and the posture indicated by the human body model at the time Tn recognized by the recognition unit 12. The probability may be calculated. For example, the transition probability from the state at time Tn to the state candidate i can be calculated by the following equation. Note that the difference in posture can be calculated as the sum of the Euclidean distances of the coordinate values of each part.

Transition probability to state candidate i =
1- (the difference between the state candidate i) / (the difference between Σ _i state candidate i)
The “difference from state candidate i” in the above expression means the difference between the posture indicated by the human body model at time Tn and the posture indicated by the coordinate value group included in the state information of state candidate i.

  Further, the state estimation unit 15 indicates the posture indicated by the state information stored in the operation pattern DB 23 for the state at each time stored in the state history table 22B and the human body model recognized by the recognition unit 12 at the corresponding time. The time change of the difference from the posture is calculated. Then, the state estimation unit 15 may calculate a transition probability when the current state is maintained and a transition probability when transitioning to the next state according to the calculated time change.

  Further, the state estimation unit 15 performs curve approximation on the coordinate values of each part at each time stored in the human body model history table 22A, and estimates the human body model at time Tn + 1 from the coordinate values on the extension line of the approximate curve. And the state estimation part 15 may calculate the transition probability to each state candidate based on the similarity with the attitude | position which the estimated human body model shows, and the attitude | position which the state information of each state candidate shows.

  The state estimation unit 15 estimates a state candidate having the highest transition probability from the state at time Tn among the state candidates as a state at time Tn + 1. For example, when the state 3, state 4, and state 11 are extracted as state candidates, and the transition probability is calculated for each state candidate as follows, the state estimation unit 15 sets the state 3 as the state at time Tn + 1. presume.

State candidate 3: Transition probability = 0.6
State candidate 4: Transition probability = 0.3
State candidate 11: transition probability = 0.1

  The state estimation unit 15 passes the state information of the estimated state to the control unit 17.

  The position estimation unit 16 refers to the human body model history table 22A of the history DB 22 and estimates the position where the object exists at the next timing based on the change in the position of the object. For example, the position estimation unit 16 performs curve approximation on the coordinate value of a predetermined part of the human body, and obtains a coordinate value on the extension line of the approximate curve and the coordinate value according to the moving speed of the predetermined part indicated by the history of the coordinate value at time Tn + 1. Estimated as the position of the object. The predetermined part of the human body may be one part or an average of a plurality of parts. Moreover, you may determine the predetermined site | part according to the technique in implementation. Furthermore, the region that can be the position of the object at time Tn + 1 may be limited according to the technique being performed. The position estimation unit 16 passes information on the position of the object at the estimated time Tn + 1 to the control unit 17.

  Based on the posture and position of the object estimated by the estimation unit 13, the control unit 17 calculates the control value of the direction and angle of view of the 3D sensor 31 at the next timing, and outputs the control value to the drive unit 32. Specifically, the control unit 17 develops the human body model having the posture indicated by the state at the time Tn + 1 estimated by the state estimation unit 15 at the position of the target at the time Tn + 1 estimated by the position estimation unit 16. The size of the human body model is corrected according to the physique of the player who is the object. The control unit 17 converts each coordinate value group of each part indicating the corrected human body model from the global coordinate system to the sensor coordinate system. As shown in FIG. 8, the control unit 17 includes a human body model converted into coordinate values in the sensor coordinate system, calculates a minimum rectangular parallelepiped region facing the 3D sensor 31, and includes 3D including this rectangular parallelepiped region. The direction and angle of view of the sensor 31 are calculated. Specifically, the control unit 17 calculates the orientation θTn + 1 (θpan_n + 1, θtilt_n + 1, θroll_n + 1) and the angle of view φTn + 1 (φxs_n + 1, φys_n + 1) of the 3D sensor 31 at time Tn + 1 as control values.

  When the drive unit 32 drives the 3D sensor 31 based on this control value, the possibility that the three-dimensional data at the time Tn + 1 can be acquired in a state where the ratio of the object in the angle of view of the 3D sensor 31 is high increases.

  The three-dimensional data acquisition apparatus 10 can be realized by a computer 40 shown in FIG. 9, for example. The computer 40 includes a central processing unit (CPU) 41, a memory 42 as a temporary storage area, and a nonvolatile storage unit 43. The computer 40 also includes an input / output device 44 including a display unit and an input unit, and a read / write (R / W) unit 45 that controls reading and writing of data with respect to the storage medium 49. The computer 40 also includes a communication interface (I / F) 46 connected to a network such as the Internet. The CPU 41, the memory 42, the storage unit 43, the input / output device 44, the R / W unit 45, and the communication I / F 46 are connected to each other via a bus 47.

  The storage unit 43 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 43 as a storage medium stores a three-dimensional data acquisition program 50 for causing the computer 40 to function as the three-dimensional data acquisition device 10. The three-dimensional data acquisition program 50 includes an acquisition process 51, a recognition process 52, an estimation process 53, and a control process 57. Further, the storage unit 43 has an information storage area 60 in which information constituting the operation pattern DB 23 is stored.

  The CPU 41 reads the three-dimensional data acquisition program 50 from the storage unit 43 and develops it in the memory 42, and sequentially executes the processes included in the three-dimensional data acquisition program 50. The CPU 41 operates as the acquisition unit 11 illustrated in FIG. 1 by executing the acquisition process 51. The CPU 41 operates as the recognition unit 12 illustrated in FIG. 1 by executing the recognition process 52. Further, the CPU 41 operates as the estimation unit 13 illustrated in FIG. 1 by executing the estimation process 53. Further, the CPU 41 operates as the control unit 17 illustrated in FIG. 1 by executing the control process 57. Further, the CPU 41 reads information from the information storage area 60 and develops the operation pattern DB 23 in the memory 42. Further, the CPU 41 stores the inputted performance schedule list 21 in the memory 42. Further, the CPU 41 creates the history DB 22 on the memory 42 when executing the recognition process 52 and the estimation process 53. Accordingly, the computer 40 that has executed the three-dimensional data acquisition program 50 functions as the three-dimensional data acquisition device 10. The CPU 41 that executes the program is hardware.

  The function realized by the three-dimensional data acquisition program 50 can be realized by, for example, a semiconductor integrated circuit, more specifically, an application specific integrated circuit (ASIC).

  Next, the operation of the three-dimensional data acquisition apparatus 10 according to the first embodiment will be described.

  The acquisition unit 11 acquires the performance schedule list 21 input in advance to the three-dimensional data acquisition apparatus 10 and stores it in a predetermined storage area. When the start of the acquisition of 3D data is instructed in accordance with the start of the performance of the target player in the state where the performance schedule list 21 is stored, the 3D data acquisition apparatus 10 displays the 3D data shown in FIG. Data acquisition processing is executed.

  In step S11, the acquisition unit 11 sets the variable n corresponding to the time to an initial value of 0.

  Next, in step S12, the acquisition unit 11 refers to the performance schedule list 21 and the technique performance history table 22C of the history DB 22, and determines whether or not the performance has ended. If the performance has not ended yet, the process proceeds to step S13, and the acquisition unit 11 acquires the three-dimensional data group PTn (Xsn, Ysn, Zsn) at time Tn from the 3D sensor 31. The acquisition unit 11 passes the acquired three-dimensional data group to the recognition unit 12. The acquisition unit 11 also acquires the orientation θTn (θpan_n, θtilt_n, θroll_n) and the angle of view φTn (φxs_n, φys_n) of the 3D sensor 31 at time Tn together with the three-dimensional data, and passes them to the estimation unit 13.

  Next, in step S14, the recognition unit 12 recognizes the human body model indicating the position and orientation of the object in the sensor coordinate system based on the three-dimensional data group delivered from the acquisition unit 11, and becomes a reference human body model. Normalize to the size of. The recognizing unit 12 stores the coordinate values body index_Tn (xs, ys, zs, ws) in the sensor coordinate system of each part constituting the human body model at time Tn in the human body model history table 22A of the history DB 22.

  Next, in step S20, a state determination process is executed. Here, the state determination processing will be described with reference to FIG.

  In step S <b> 21, the state determination unit 14 receives the coordinate value group of the sensor coordinate system at time Tn stored in the human body model history table 22 </ b> A as the information on the orientation and angle of view of the 3D sensor 31 passed from the acquisition unit 11. To convert to a coordinate value group in the global coordinate system. The state determination unit 14 updates the coordinate value at time Tn stored in the human body model history table 22A to the coordinate value converted into the global coordinate system.

  Next, in step S22, the state determination unit 14 refers to the state history table 22B to determine whether or not the state at the time Tn-1 indicates a technique break posture. In the case of a state indicating the posture of a technique, the process proceeds to step S23, and in the case of a state indicating an attitude in the middle of the technique, the process proceeds to step S24.

  In step S23, the state determination unit 14 refers to the performance schedule list 21 and the technique execution history table 22C, and specifies the next technique after the completed technique. Then, the state determination unit 14 acquires an operation pattern including the specified technique from the operation pattern DB 23, and in each of the acquired operation patterns, the state indicating the first posture (the state next to the state indicating the posture of the technique delimiter) ) Is extracted as a state candidate at time Tn.

  On the other hand, in step S24, the state determination unit 14 extracts states that can transition from the state at time Tn-1 as state candidates at time Tn in the operation pattern including the state at time Tn-1.

  Next, in step S25, the state determination unit 14 selects a state candidate indicating the posture most similar to the posture of the human body model stored in the human body model history table 22A from the state information of each of the extracted state candidates. It is determined as the state at Tn. The state determination unit 14 stores the state number of the determined state in association with the time Tn together with the pattern number of the operation pattern including the state in the state history table 22B. Then, the process returns to the three-dimensional data acquisition process shown in FIG.

  Next, a state estimation process is performed at step S30. Here, the state estimation process will be described with reference to FIG.

  In steps S31 to S33, the state estimation unit 15 reads the time Tn-1 as the time Tn and the time Tn as the time Tn + 1, which is the same processing as the processing in steps S22 to S24 of the state determination processing (FIG. 11). The process is executed to extract a state candidate at time Tn + 1.

  Next, in step S34, the state estimation unit 15 calculates a transition probability from the state at time Tn for each of the extracted state candidates. The state estimation unit 15 estimates a state candidate having the highest transition probability from the state at time Tn among the state candidates as a state at time Tn + 1. The state estimation unit 15 passes the state information of the estimated state to the control unit 17. Then, the process returns to the three-dimensional data acquisition process shown in FIG.

  Next, in step S41, the position estimation unit 16 refers to the human body model history table 22A of the history DB 22, and estimates the position where the object exists at time Tn + 1 based on the change in the position of the object. The position estimation unit 16 passes information on the position of the object at the estimated time Tn + 1 to the control unit 17.

  Next, in step S <b> 42, the control unit 17 develops the human body model having the posture indicated by the state at the time Tn + 1 estimated by the state estimation unit 15 at the position of the target at the time Tn + 1 estimated by the position estimation unit 16. . And the control part 17 correct | amends the size of a human body model according to the physique of the player who is a target object. Next, in step S43, the control unit 17 converts each coordinate value group of each part indicating the corrected human body model from the global coordinate system to the sensor coordinate system.

  Next, in step S <b> 44, the control unit 17 calculates a minimum rectangular parallelepiped region that includes the human body model converted into the coordinate value of the sensor coordinate system and faces the 3D sensor 31. Next, in step S45, the control unit 17 calculates the orientation θTn + 1 (θpan_n + 1, θtilt_n + 1, θroll_n + 1) and the angle of view Tn + 1 (φxs_n + 1, φys_n + 1) of the 3D sensor 31 including this rectangular parallelepiped region as control values. The control unit 17 outputs the calculated control value of the direction and the angle of view of the 3D sensor 31 at the calculated time Tn + 1 to the drive unit 32.

  Next, in step S46, the acquisition unit 11 increments the variable n by 1, and the process returns to step S12. If it is determined in step S12 that the performance has ended, the three-dimensional data acquisition process ends.

  As described above, the three-dimensional data acquisition apparatus 10 according to the first embodiment includes the performance schedule list input in advance, the motion pattern expressing the transition of the posture, and the current position and posture of the target object. Based on this, the position and orientation of the object at the next timing are estimated. Then, the three-dimensional data acquisition apparatus 10 controls the angle of view and orientation of the 3D sensor so that the object having the estimated position and orientation is accommodated to the full angle of view. Thereby, even if it is the target object whose operation direction changes suddenly, acquisition of three-dimensional data can be continued in the state where the ratio of the target object to the angle of view of the 3D sensor is high.

  In addition, when estimating the state at the next timing, the three-dimensional data acquisition apparatus 10 associates a plurality of operations including the same posture with each posture corresponding to one state, and connects the transitionable states. An operation pattern represented by a tree structure is used. Examples of the plurality of actions including the same posture include an action indicating a reference technique, an action indicating a technique lower than the reference technique, and an action when the reference technique and the lower technique are not established or failed. Thereby, even when the motion of the player, which is the object, becomes an unscheduled motion, the posture of the object at the next timing can be accurately estimated, and the proportion of the object in the angle of view of the 3D sensor is Acquisition of three-dimensional data can be continued in a high state.

  In addition, by estimating the posture indicated by the state having the highest transition probability among the states included in the motion pattern as the posture of the object at the next timing, the next timing can be more accurately performed. The posture of the object at can be estimated.

Second Embodiment
Next, a second embodiment will be described. In the three-dimensional data acquisition device according to the second embodiment, the same parts as those of the three-dimensional data acquisition device 10 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  As illustrated in FIG. 1, the three-dimensional data acquisition apparatus 210 includes an acquisition unit 11, a recognition unit 12, an estimation unit 213, and a control unit 17. In addition, in a predetermined storage area of the three-dimensional data acquisition apparatus 210, a performance schedule list 21, a history DB 22, and an operation pattern DB 223 are stored.

  In addition to the tree structure representing the operation pattern included in the operation pattern DB 23 according to the first embodiment and the state information of each state, the operation pattern DB 223 includes a connection operation table 223A as shown in FIG. Included in the scheduled performance list 21 such as walking, stepping, posing, performing tricks such as forward rotation, etc. when the preceding technique is finished and the subsequent technique is carried out. Some operations may be performed. In such a connecting operation, when the preceding technique is the connection source technique and the subsequent technique is the connection destination technique, there is a tendency that an operation of a pattern determined to some extent is performed depending on the type of the connected technique. . Therefore, the joining operation table 223A stores the joining operation pattern in association with the connection source technique and the connection destination technique.

  For example, in the example of FIG. 13, the connection operation pattern with the connection operation pattern ID = 1 represents that the technique 5 is performed as a connection operation between the technique 4 and the technique 6. The connection operation pattern may include a plurality of connection operations, such as a connection operation pattern of connection operation pattern ID = 3, 4, and 5. In the example of FIG. 13, the connection operation pattern in which the “connection operation pattern” field is blank represents that the operations in the “connection source technique” and “connection destination technique” fields are repeated a plurality of times. For example, when the connection operation pattern ID = 3 and 6 is implemented as the connection operation pattern between the technique 6 and the technique 1, the technique 6 → the connection operation A → the connection operation A → the connection operation B → the technique 1 is performed.

  Further, the operation pattern stored in the operation pattern DB 223 includes an operation pattern of a linkage operation.

  The estimation unit 213 includes a state determination unit 214, a state estimation unit 215, and a position estimation unit 16.

  Similarly to the state determination unit 14 of the first embodiment, the state determination unit 214 determines which state the posture of the target human body model recognized by the recognition unit 12 includes in which operation pattern. When the state at the time Tn−1 is a state indicating the posture of the technique, the state determination unit 214 extracts a state candidate at the time Tn from the motion pattern including the technique next to the technique that has been completed. At this time, as the next technique, the state determination unit 214 refers to not only the next technique obtained from the acting schedule list 21 but also the connection technique table 223A, the technique completed at the time Tn-1 and the next technique. This also applies to the connection between the two.

  For example, according to the performance schedule list 21 shown in FIG. 2, when the second technique 6 in the performance schedule list is completed at time Tn−1, the state determination unit 214 operates the operation pattern ( For example, state candidates are extracted from pattern 1) in FIG. Furthermore, the state determination unit 214 refers to, for example, the connection operation table 223A illustrated in FIG. 13 and extracts a state candidate from the operation pattern of the connection operation A included in the connection operation pattern of the technique 6 and the technique 1.

  The state estimation unit 215 differs from the state estimation unit 15 of the first embodiment in that the state estimation unit 215 refers to the connection operation table 223A as well as the state determination unit 214 when extracting the state candidate at the time Tn + 1.

  The three-dimensional data acquisition device 210 can be realized by a computer 40 shown in FIG. 9, for example. The storage unit 43 of the computer 40 stores a three-dimensional data acquisition program 250 for causing the computer 40 to function as the three-dimensional data acquisition device 210. The three-dimensional data acquisition program 250 includes an acquisition process 51, a recognition process 52, an estimation process 253, and a control process 57. The storage unit 43 has an information storage area 60 in which information constituting the operation pattern DB 223 is stored.

  The CPU 41 reads the three-dimensional data acquisition program 250 from the storage unit 43 and expands it in the memory 42, and sequentially executes the processes included in the three-dimensional data acquisition program 250. The CPU 41 operates as the estimation unit 213 illustrated in FIG. 1 by executing the estimation process 253. Other processes are the same as those of the three-dimensional data acquisition program 50 in the first embodiment. As a result, the computer 40 that has executed the three-dimensional data acquisition program 250 functions as the three-dimensional data acquisition device 210.

  The function realized by the three-dimensional data acquisition program 250 can be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC.

  Next, the operation of the three-dimensional data acquisition apparatus 210 according to the second embodiment will be described. Also in the second embodiment, the three-dimensional data acquisition process shown in FIG. 10 is executed, but the following points are different from the three-dimensional data acquisition process in the first embodiment. The first point is that the connection operation table 223A is also referred to when the state candidate at time Tn is extracted in step S23 of the state determination process shown in FIG. The second point is that the connection operation table 223A is also referred to when the state candidate of the item Tn + 1 is extracted in step S32 of the state estimation process shown in FIG. About another point, since the process similar to the three-dimensional data acquisition process in 1st Embodiment is performed, detailed description is abbreviate | omitted.

  As described above, the three-dimensional data acquisition apparatus 210 according to the second embodiment determines the state at the time Tn at the timing of shifting to the next technique and estimates the state at the time Tn + 1. Also consider the connection between tricks. Thereby, even when an operation not described in the performance schedule list is performed between the techniques, the acquisition of the three-dimensional data is continued in a state where the ratio of the object in the angle of view of the three-dimensional sensor is high. be able to.

  In addition, although each said embodiment demonstrated the case where the gymnast who performs acting was made into a target object, it is not limited to this. It is also possible to apply an object that performs an action according to a prior plan, for example, a figure skating, a half pipe such as a skateboard or a snowboard, a mogul, a swimming jump, a performance, a dance, or the like.

  In each of the above embodiments, the size of the human body model recognized by the recognition unit is normalized and then compared with the state information (coordinate group of each part indicating posture) stored in the motion pattern DB. Although described, it is not limited to this. For example, the human body model indicated by the state information stored in the motion pattern DB is corrected to a size corresponding to the actual player size, and then compared with the human body model whose size recognized by the recognition unit is not normalized. You may make it do. As shown in FIG. 14, in order to apply to various sizes by normalizing or correcting the size of the human body model, it is necessary to store a plurality of state information for one state in the operation pattern DB. Absent.

  Moreover, although each said embodiment demonstrated the case where the state candidate with the highest transition probability from the state of the time Tn among the state candidates of the time Tn + 1 was estimated as a state of the time Tn + 1, it is not limited to this. A plurality of state candidates may be adopted as the state at time Tn + 1, such as all of the extracted state candidates or state candidates having a transition probability equal to or higher than a predetermined value. In this case, a region including all the postures indicated by each of the adopted states can be calculated as a region occupied by the object at time Tn + 1.

  In each of the above embodiments, the case where the history of the coordinate values of each part constituting the human body model is referred to when estimating the position of the object at time Tn + 1 is not limited to this. For example, when the posture indicated by the estimated state at time Tn + 1 is a posture in which a predetermined part of the human body is at a fixed point, the position based on that part may be estimated. The predetermined part of the human body is, for example, the center of gravity of a hand supporting the body in the case of a pommel horse, the hand or foot that is on the ground in the case of the floor, or the spine in the case of the human body in the air.

  In the above description, the mode in which the three-dimensional data acquisition programs 50 and 250 are stored (installed) in the storage unit 43 in advance has been described. However, the present invention is not limited to this. The program can also be provided in a form stored in a storage medium such as a CD-ROM, DVD-ROM, or USB memory.

  Regarding the above embodiments, the following additional notes are disclosed.

(Appendix 1)
An acquisition unit for acquiring three-dimensional data indicating a three-dimensional position of each point on the object detected by the three-dimensional sensor;
A recognition unit for recognizing the position and orientation of the object based on the three-dimensional data acquired by the acquisition unit;
With reference to an operation pattern storage unit that stores a plurality of operation patterns defined by state transitions indicating a plurality of related operations including the same posture and indicating the posture of the object, the received object is Based on the information of the operation scheduled to be performed and the posture of the target object recognized by the recognition unit, the posture of the target object at the next timing is estimated, and the estimated posture is recognized by the recognition unit. An estimation unit for estimating a position where the object is present at the next timing based on the position change of the object performed,
A control unit that controls the orientation and angle of view of the three-dimensional sensor based on the posture and position of the object estimated by the estimation unit;
A three-dimensional data acquisition device.

(Appendix 2)
The motion pattern includes a plurality of related operations including the same posture, an operation indicating a standard technique, and an operation indicating another technique that can be transitioned from the posture of an object during the execution of the standard technique, It is an operation pattern showing an operation of an object including an operation when the reference technique and the other technique are not established or failed,
The three-dimensional data acquisition apparatus according to claim 1, wherein the estimation unit refers to a list of techniques scheduled to be performed by the target object as information on an operation scheduled to be performed by the target object.

(Appendix 3)
In the motion pattern defined including the state corresponding to the posture of the object recognized by the recognition unit, the estimation unit indicates the posture indicated by the state that can be transitioned from the corresponding state at the next timing. The three-dimensional data acquisition device according to supplementary note 1 or supplementary note 2, which estimates the posture of the object.

(Appendix 4)
The recognition unit recognizes a human body model represented by a set of coordinate values of each part of the object based on the three-dimensional data acquired by the acquisition unit;
The movement pattern is defined by a state transition indicating a posture of the object represented by a set of coordinate values of each part of the object,
The estimation unit is recognized by the recognition unit in at least one coordinate system of a relative coordinate system of each part of the object, a global coordinate system, and a coordinate system based on a movement direction indicated by the movement pattern. In addition, the state of the motion pattern represented by a set of coordinate values whose difference from the set of coordinate values representing the human body model is within a predetermined range corresponds to the posture of the object recognized by the recognition unit. The three-dimensional data acquisition device according to attachment 3, wherein the three-dimensional data acquisition device is determined as a state.

(Appendix 5)
The three-dimensional data acquisition device according to supplementary note 3 or supplementary note 4, wherein the estimation unit estimates a state that can transition from the corresponding state based on a transition probability between states.

(Appendix 6)
The transition probability includes any one of a difficulty level of movement, a characteristic of the target object, a speed of change of the target object attitude, and a deviation degree from the current object attitude and the attitude indicated by the corresponding state. 3D data acquisition device.

(Appendix 7)
The operation pattern storage unit further stores a connection operation pattern indicating an operation between techniques,
According to the list of techniques to be performed, the estimation unit includes a motion pattern including a technique to be performed next, and a linkage operation indicating an operation between a technique that has just been completed and a technique to be performed next. The three-dimensional data acquisition device according to any one of appendix 2 to appendix 6, which refers to a pattern.

(Appendix 8)
Obtaining three-dimensional data indicating the three-dimensional position of each point on the object detected by the three-dimensional sensor;
Based on the acquired three-dimensional data, recognize the position and orientation of the object,
With reference to an operation pattern storage unit that stores a plurality of operation patterns defined by state transitions indicating a plurality of related operations including the same posture and indicating the posture of the object, the received object is Based on the information on the action to be performed and the posture of the recognized object, the posture of the object at the next timing is estimated, and the estimated posture and the change in the position of the recognized object On the basis of the position of the object at the next timing,
A three-dimensional data acquisition method in which a computer executes processing including controlling the orientation and angle of view of the three-dimensional sensor based on the estimated posture and position of the object.

(Appendix 9)
The motion pattern includes a plurality of related operations including the same posture, an operation indicating a standard technique, and an operation indicating another technique that can be transitioned from the posture of an object during the execution of the standard technique, It is an operation pattern showing an operation of an object including an operation when the reference technique and the other technique are not established or failed,
The three-dimensional data acquisition method according to supplementary note 8, wherein a list of techniques scheduled to be performed by the target object is referred to as information on an operation scheduled to be performed by the target object.

(Appendix 10)
Supplementary Note 8: Estimating a posture indicated by a state capable of transitioning from the corresponding state as a posture of the target object at the next timing in an operation pattern defined including a state corresponding to the recognized posture of the target object Alternatively, the three-dimensional data acquisition method according to attachment 9.

(Appendix 11)
Based on the acquired three-dimensional data, recognize a human body model represented by a set of coordinate values of each part of the object,
The movement pattern is defined by a state transition indicating a posture of the object represented by a set of coordinate values of each part of the object,
A set of coordinate values representing the recognized human body model in at least one coordinate system of a relative coordinate system of each part of the object, a global coordinate system, and a coordinate system based on the moving direction of the motion indicated by the motion pattern The three-dimensional data acquisition method according to supplementary note 10, wherein the state of the motion pattern represented by a set of coordinate values within which a difference between and is within a predetermined range is determined as a state corresponding to the recognized posture of the object.

(Appendix 12)
The three-dimensional data acquisition method according to supplementary note 10 or supplementary note 11, wherein a state that can transition from the corresponding state is estimated based on a transition probability between states.

(Appendix 13)
The transition probability includes any one of a difficulty level of motion, a characteristic of the target object, a speed of change of the target object attitude, and a deviation degree from the current object attitude and the attitude indicated by the corresponding state. 3D data acquisition method.

(Appendix 14)
The operation pattern storage unit further stores a connection operation pattern indicating an operation between techniques,
In accordance with the list of techniques to be executed, the operation pattern including the technique to be executed next is referred to, and the connection operation pattern indicating the operation between the technique just completed and the technique to be executed next is referred to. The three-dimensional data acquisition method according to any one of appendix 9 to appendix 13.

(Appendix 15)
Obtaining three-dimensional data indicating the three-dimensional position of each point on the object detected by the three-dimensional sensor;
Based on the acquired three-dimensional data, recognize the position and orientation of the object,
With reference to an operation pattern storage unit that stores a plurality of operation patterns defined by state transitions indicating a plurality of related operations including the same posture and indicating the posture of the object, the received object is Based on the information on the action to be performed and the posture of the recognized object, the posture of the object at the next timing is estimated, and the estimated posture and the change in the position of the recognized object On the basis of the position of the object at the next timing,
A three-dimensional data acquisition program for causing a computer to execute processing including controlling the orientation and angle of view of the three-dimensional sensor based on the estimated posture and position of the object.

(Appendix 16)
The motion pattern includes a plurality of related operations including the same posture, an operation indicating a standard technique, and an operation indicating another technique that can be transitioned from the posture of an object during the execution of the standard technique, It is an operation pattern showing an operation of an object including an operation when the reference technique and the other technique are not established or failed,
The three-dimensional data acquisition program according to supplementary note 15, wherein a list of techniques scheduled to be performed by the object is referred to as information on an operation scheduled to be performed by the object.

(Appendix 17)
(Supplementary Note 15) In the motion pattern defined including the state corresponding to the recognized posture of the object, the posture indicated by the state capable of transition from the corresponding state is estimated as the posture of the object at the next timing. Or the three-dimensional data acquisition program according to attachment 16.

(Appendix 18)
Based on the acquired three-dimensional data, recognize a human body model represented by a set of coordinate values of each part of the object,
The movement pattern is defined by a state transition indicating a posture of the object represented by a set of coordinate values of each part of the object,
A set of coordinate values representing the recognized human body model in at least one coordinate system of a relative coordinate system of each part of the object, a global coordinate system, and a coordinate system based on the moving direction of the motion indicated by the motion pattern The three-dimensional data acquisition program according to appendix 17, wherein the state of the motion pattern represented by a set of coordinate values that is within a predetermined range is determined as a state corresponding to the recognized posture of the object.

(Appendix 19)
The three-dimensional data acquisition program according to supplementary note 17 or supplementary note 18, wherein a transitionable state from the corresponding state is estimated based on a transition probability between states.

(Appendix 20)
The transition probability includes any one of a difficulty level of movement, a characteristic of the target object, a speed of change of the target object attitude, and a deviation degree from the attitude indicated by the current object attitude and the corresponding state. 3D data acquisition program.

DESCRIPTION OF SYMBOLS 10, 210 Three-dimensional data acquisition apparatus 11 Acquisition part 12 Recognition part 13, 213 Estimation part 14, 214 State determination part 15, 215 State estimation part 16 Position estimation part 17 Control part 21 Scheduled action list 22 History DB
22A Human body model history table 22B State history table 22C Technique implementation history table 23, 223 Operation pattern DB
223A Connection operation table 31 3D sensor 32 Drive unit 40 Computer 41 CPU
42 Memory 43 Storage 49 Storage medium 50, 250 Three-dimensional data acquisition program

Claims (8)

An acquisition unit for acquiring three-dimensional data indicating a three-dimensional position of each point on the object detected by the three-dimensional sensor;
A recognition unit for recognizing the position and orientation of the object based on the three-dimensional data acquired by the acquisition unit;
With reference to an operation pattern storage unit that stores a plurality of operation patterns defined by state transitions indicating a plurality of related operations including the same posture and indicating the posture of the object, the received object is Based on the information of the operation scheduled to be performed and the posture of the target object recognized by the recognition unit, the posture of the target object at the next timing is estimated, and the estimated posture is recognized by the recognition unit. An estimation unit for estimating a position where the object is present at the next timing based on the position change of the object performed,
A control unit that controls the orientation and angle of view of the three-dimensional sensor based on the posture and position of the object estimated by the estimation unit;
A three-dimensional data acquisition device. The motion pattern includes a plurality of related operations including the same posture, an operation indicating a standard technique, and an operation indicating another technique that can be transitioned from the posture of an object during the execution of the standard technique, It is an operation pattern showing an operation of an object including an operation when the reference technique and the other technique are not established or failed,
The three-dimensional data acquisition apparatus according to claim 1, wherein the estimation unit refers to a list of techniques scheduled to be performed by the object as information on an operation scheduled to be performed by the object.   In the motion pattern defined including the state corresponding to the posture of the object recognized by the recognition unit, the estimation unit indicates the posture indicated by the state that can be transitioned from the corresponding state at the next timing. The three-dimensional data acquisition apparatus according to claim 1, wherein the three-dimensional data acquisition apparatus estimates the posture of the object. The recognition unit recognizes a human body model represented by a set of coordinate values of each part of the object based on the three-dimensional data acquired by the acquisition unit;
The movement pattern is defined by a state transition indicating a posture of the object represented by a set of coordinate values of each part of the object,
The estimation unit is recognized by the recognition unit in at least one coordinate system of a relative coordinate system of each part of the object, a global coordinate system, and a coordinate system based on a movement direction indicated by the movement pattern. In addition, the state of the motion pattern represented by a set of coordinate values whose difference from the set of coordinate values representing the human body model is within a predetermined range corresponds to the posture of the object recognized by the recognition unit. The three-dimensional data acquisition device according to claim 3, which is determined as a state.   The three-dimensional data acquisition apparatus according to claim 3, wherein the estimation unit estimates a state that can transition from the corresponding state based on a transition probability between states.   The transition probability includes any one of a difficulty level of movement, a characteristic of the target object, a speed of change of the target object attitude, and a deviation degree from a current attitude of the target object and a corresponding attitude. The three-dimensional data acquisition device described. The operation pattern storage unit further stores a connection operation pattern indicating an operation between techniques,
According to the list of techniques to be performed, the estimation unit includes a motion pattern including a technique to be performed next, and a linkage operation indicating an operation between a technique that has just been completed and a technique to be performed next. The three-dimensional data acquisition apparatus according to claim 2, wherein the pattern is referred to. Obtaining three-dimensional data indicating the three-dimensional position of each point on the object detected by the three-dimensional sensor;
Based on the acquired three-dimensional data, recognize the position and orientation of the object,
With reference to an operation pattern storage unit that stores a plurality of operation patterns defined by state transitions indicating a plurality of related operations including the same posture and indicating the posture of the object, the received object is Based on the information on the action to be performed and the posture of the recognized object, the posture of the object at the next timing is estimated, and the estimated posture and the change in the position of the recognized object On the basis of the position of the object at the next timing,
A three-dimensional data acquisition method in which a computer executes processing including controlling the orientation and angle of view of the three-dimensional sensor based on the estimated posture and position of the object.