JP6704606B2 - Judgment system and judgment method - Google Patents

Description

The present invention relates to a determination system and a determination method.

In recent years, in sports, it is necessary to record and manage play, and a determination system for determining play is required. There is also known a technique of determining the behavior of the target person from the characteristics of the movement trajectory of the target person generated from image information captured by a camera or the like (for example, see Patent Document 1).

JP, 2011-100175, A

However, if an attempt is made to apply the above-described technique for determining a behavior to a determination system for determining a play, for example, in order to deal with the case where the players overlap each other in the image information, a plurality of images captured from a plurality of directions are captured. Image information needs to be processed. In addition, when there are a plurality of players in the used image information, it is necessary to generate movement loci corresponding to the plurality of players. Therefore, in the conventional determination system, the processing load is high, and it is difficult to accurately determine the play of the target person while suppressing the processing load.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a determination system and a determination method capable of highly accurately performing a play determination of a target person while suppressing a processing load. ..

In order to solve the above problems, one embodiment of the present invention estimates a skeleton or a joint of a target person based on at least a detection device capable of detecting shape and position information of an object and the detection information detected by the detection device. A skeleton estimation unit, a trajectory generation unit that generates a movement trajectory of the target person based on the detection information detected by the detection device, a past movement trajectory of the target person, and a skeleton or joint of the target person. A determiner generated so as to determine the play of the target person that indicates the motion and behavior of the target person based on the above, a skeleton or joint of the target person estimated by the skeleton estimation unit, and the trajectory generation unit And a play determination unit that determines the play of the target person based on the movement trajectory of the target person , the detection device includes a laser radar device and an imaging device, the skeleton estimation unit, Based on the contour of the target person detected by the laser radar device and image information including the target person captured by the imaging device, the skeleton or joint of the target person is estimated, and the trajectory generation unit is a laser radar. The first position information, which is the position information of the target person in the target area based on the detection by the device, and the second position information, which is the position information of the target person in the target area based on the image information, are mutually complemented. The determination system is characterized in that position information of the target person is corrected, and a movement locus of the target person is generated based on the corrected position information of the target person .

Further, according to an aspect of the present invention, in the above determination system, the trajectory generation unit generates a movement trajectory of an object existing around the target person based on the detection information, and the play determination unit is The play of the target person is determined from the movement track of the target person, the skeleton or joint of the target person, and the movement track of the target person by using the determiner.

Further, according to an aspect of the present invention, in the above determination system, the target person is identified based on a partial image in the image information corresponding to the first position information, and the identified target person is An identification processing unit for providing identification information for identifying a target person, wherein the trajectory generation unit is configured to generate a movement trajectory of the target person in association with the identification information provided by the identification processing unit. To do.

Further, according to one aspect of the present invention, a skeleton estimating step of estimating a skeleton or a joint of the target person based on detection information detected by a detection device capable of detecting at least the shape and position information of the object, A locus generation unit generates a movement locus of the target person based on the detection information detected by the detection device, and a play determination unit includes a movement locus of the target person in the past and the movement of the target person. A determiner that is generated to determine the play of the target person that indicates the motion and behavior of the target person based on a skeleton or joint; a skeleton or joint of the target person estimated by the skeleton estimating unit; on the basis of the moving track of the object person trajectory generation unit is generated, the saw including a play determining step of determining the play of the target person, the detection device is included laser radar apparatus and an imaging apparatus, wherein In the skeleton estimating step, the skeleton estimating unit determines the skeleton or joint of the target person based on the contour of the target person detected by the laser radar device and the image information including the target person captured by the imaging device. In the trajectory generating step, the trajectory generating unit estimates the first position information that is position information of the target person in the target area based on detection by the laser radar device, and the target in the target area based on the image information. Compensating the position information of the target person by mutually complementing with the second position information that is the position information of the person, and generating a movement locus of the target person based on the corrected position information of the target person. Is a determination method.

According to the present invention, it is possible to highly accurately determine the play of the target person while suppressing the processing load.

It is a block diagram which shows an example of the position tracking system by 1st Embodiment. It is a figure which shows the installation example of the laser radar apparatus and imaging device in 1st Embodiment. It is a figure which shows the data example of the target object information storage part in 1st Embodiment. It is a figure which shows the example of data of the movement track storage part in 1st Embodiment. It is a figure which shows the data example of the target object locus|storage part in 1st Embodiment. It is a figure explaining an example of generation processing of a movement locus in a 1st embodiment. It is a figure explaining an example of calculation processing of three-dimensional coordinates in a 1st embodiment. It is a figure explaining an example of identification processing of the target person in a 1st embodiment. It is a flow chart which shows an example of operation of the position tracking system by a 1st embodiment. It is a block diagram which shows an example of the play determination system by 2nd Embodiment. It is a flowchart which shows an example of operation|movement of the play determination system by 2nd Embodiment. It is a figure which shows an example of the display screen in the modification of the play determination system by 2nd Embodiment. It is a figure which shows another modification of the play determination system by 2nd Embodiment.

Hereinafter, a position tracking system (position measuring system) and a play determination system according to embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing an example of a position tracking system 1 according to the first embodiment.
As shown in FIG. 1, the position tracking system 1 includes an information processing device 10, a laser radar device 20, and an imaging device 30. The position tracking system 1 is an example of a position measuring system.

The laser radar device 20 (an example of a detection device) irradiates a target area with laser light (for example, laser light such as near-infrared rays) and includes at least a distance and a direction to the object based on reflected light from the object. It is a lidar device that detects detection information. Here, the target area is, for example, a court, a stadium, or the like, which is a stadium for playing sports. The laser radar device 20 detects an object in the target area based on the reflected light from the object, and in addition to the distance and direction to the object, various information such as the contour of the object and the shape of the surface of the object. To detect. The laser radar device 20 uses, for example, a TOF (Time of Flight) method that detects the distance to the object from the moving time of the laser light from the irradiation of the laser light to the reception of the reflected light reflected by the object. To detect. The laser radar device 20 outputs the detected detection information to the information processing device 10.

The imaging device 30 is, for example, a camera device, and captures an image including the target area. The imaging device 30 outputs image information representing the captured image to the information processing device 10.
For convenience of explanation, one laser radar device 20 and one imaging device 30 are shown in FIG. 1, but the position tracking system 1 includes a plurality of laser radar devices 20 and a plurality of imaging devices 30. May be provided.

FIG. 2 is a diagram showing an installation example of the laser radar device 20 and the imaging device 30 in the present embodiment.
As shown in FIG. 2, a plurality of laser radar devices 20 and a plurality of imaging devices 30 are arranged so that the three-dimensional position coordinates of an object (for example, a player PL1) in the court AR1, which is an object area, can be detected. There is. In the present embodiment, the three-dimensional position coordinates are, for example, the X axis and the Y axis that are orthogonal to the plane of the coat AR1, and the height direction is the Z axis.

The information processing device 10 is, for example, a computer device such as a PC (personal computer). The information processing device 10 generates position information of the target object in the target area based on the detection information detected by the laser radar device 20 and the image information captured by the imaging device 30, and based on the generated position information, Generate a movement trajectory of the object. Here, the target object is, for example, a target person such as a player or a player, and a target object indicating a tool such as a ball, a racket, a net, or a goal.
The information processing device 10 also includes a storage unit 11 and a control unit 12.

The storage unit 11 stores information used for various processes of the information processing device 10. The storage unit 11 includes a radar detection storage unit 111, an image information storage unit 112, an object information storage unit 113, a movement trajectory storage unit 114, and an object trajectory storage unit 115.

The radar detection storage unit 111 stores the detection information detected by the laser radar device 20 and the time information in association with each other. In the radar detection storage unit 111, the detection information acquired from the laser radar device 20 by the data acquisition unit 121 described later is stored in synchronization with the image information captured by the imaging device 30.

The image information storage unit 112 stores the image information captured by the image capturing apparatus 30 and the time information in association with each other. The image information storage unit 112 stores the image information acquired from the imaging device 30 by the data acquisition unit 121 described later in synchronization with the detection information detected by the laser radar device 20.

The target object information storage unit 113 stores information for identifying a target object that is a person or an object. Note that the target person here also includes a target person. Here, an example of data stored in the object information storage unit 113 will be described with reference to FIG.
FIG. 3 is a diagram showing an example of data in the target object information storage unit 113 in this embodiment.
As illustrated in FIG. 3, the target object information storage unit 113 stores, for example, the “target object name” and the “feature amount information” in association with each other.

Here, the “object name” is a name of the object or the like and is identification information for identifying the object. Further, the “feature amount information” indicates, for example, a feature amount for identifying the target object from the image information, and is, for example, a shape, a color, a size, an image pattern or the like for determining the target object. In addition, the “feature amount information” may be a feature amount for face recognition, a feature amount of clothing such as a uniform, and the like when the object is a target person such as a specific player.
In the example illustrated in FIG. 3, the “object name” is stored in the object information storage unit 113 in association with “player A” and the feature amount information of the “player A”. The “object name” is stored in association with the “ball” and the feature amount information of the “ball”.

Returning to the description of FIG. 1, the movement trajectory storage unit 114 stores the movement trajectory information of the detected object. The movement track storage unit 114 stores, for example, identification information of the detected object, time information, and three-dimensional position coordinates in association with each other. Here, with reference to FIG. 4, an example of data stored in the movement track storage unit 114 will be described.

FIG. 4 is a diagram showing an example of data in the movement locus storage unit 114 in this embodiment.
As shown in FIG. 4, the movement locus storage unit 114 stores the “identification ID”, the “time”, and the “coordinates” in association with each other. Here, the “identification ID” is, for example, identification information for identifying an object detected by the laser radar device 20, and the “time” indicates time information. Note that the time information may include date information. The “coordinates” are three-dimensional position coordinates, for example, two-dimensional coordinates in the target area as shown in FIG.
The example shown in FIG. 4 indicates that the movement locus with the “identification ID” of “ID1” is stored in association with the “time” and the “coordinates”.

Returning to the description of FIG. 1 again, the object trajectory storage unit 115 stores the identification information of the object identified by the identification processing unit 123, which will be described later, and the movement trajectory of the object in association with each other. Here, with reference to FIG. 5, an example of data stored in the object trajectory storage unit 115 in the present embodiment will be described.

FIG. 5 is a diagram showing an example of data in the object trajectory storage unit 115 in this embodiment.
In FIG. 5, the object trajectory storage unit 115 stores “object name”, “time”, and “coordinate” in association with each other. Here, the information in which the “time” and the “coordinates” are associated with each other corresponds to the above-described movement trajectory.
The example shown in FIG. 5 indicates that the movement locus whose “object name” corresponds to “player A” is stored in association with “time” and “coordinates”. Note that, in the example shown in FIG. 5, the “object name” is “player A” and indicates the movement trajectory corresponding to the object person, but the object trajectory storage unit 115 stores, for example, a ball or a bat. The movement locus corresponding to the tool (target object) such as may be stored.

Returning to the description of FIG. 1 again, the control unit 12 is, for example, a processor including a CPU (Central Processing Unit) and controls the information processing apparatus 10 in a centralized manner. The control unit 12, for example, position information (first position information) of the target object in the target area based on the detection information detected by the laser radar device 20 and the target object in the target area based on the image information captured by the imaging device 30. The position information (second position information) and the position information (second position information) are mutually complemented to generate a movement trajectory of the object.
The control unit 12 includes a data acquisition unit 121, an image processing unit 122, an identification processing unit 123, and a trajectory generation unit 124.

The data acquisition unit 121 (an example of an acquisition unit) acquires and synchronizes the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. The data acquisition unit 121 stores the detection information acquired from the laser radar device 20 and the time information in the radar detection storage unit 111 in association with each other. Further, the data acquisition unit 121 stores the image information acquired from the imaging device 30 and the time information in the image information storage unit 112 in association with each other. The data acquisition unit 121 synchronizes the detection information and the image information by adding time information to the radar detection storage unit 111 and the image information storage unit 112.

The image processing unit 122 executes various image processes based on the image information acquired by the data acquisition unit 121. The image processing unit 122 generates position information (second position information) of the target object in the target area based on the image information, for example. The position information in the target area here is, for example, coordinate information of the center of gravity of the target object. Further, the image processing unit 122 extracts, for example, a partial image corresponding to the object detected by the laser radar device 20 from the image information, and based on the feature amount stored in the object information storage unit 113 for the partial image. Then, the image recognition process for identifying the object is executed. The image processing unit 122 executes, for example, face recognition processing based on the feature amount, pattern recognition processing based on the feature amount, and the like.

The identification processing unit 123 identifies the object as a target object based on the partial image in the image information corresponding to the position of the object detected based on the detection information acquired by the data acquisition unit 121, and identifies the identified target object. On the other hand, identification information for identifying the object is added. The identification processing unit 123 extracts, for example, a partial image in the image information based on the contour of the object detected by the laser radar device 20, and identifies the object as an object based on the extracted partial image. Specifically, the identification processing unit 123 causes the image processing unit 122 to extract a partial image based on the contour of the object and to perform image processing such as face recognition on the partial image. The identification processing unit 123 identifies an object as an object based on the result of the image processing by the image processing unit 122, and adds identification information for identifying the object to the identified object.

The trajectory generation unit 124 (an example of a position measurement unit), based on the detection information (for example, the distance and direction to the object) detected by the laser radar device 20, the position information (first position information) of the target object in the target area. To generate. The locus generation unit 124 generates a movement locus of the object on which the object has moved based on the generated first position information and the above-described second position information. That is, the trajectory generation unit 124 mutually complements the first position information and the second position information, corrects the position information of the target object, and moves the target object based on the corrected position information of the target object. Generate a trajectory. Here, with reference to FIG. 6, a process of generating a movement locus by mutually complementing the first position information and the second position information by the locus generation unit 124 will be described.

FIG. 6 is a diagram illustrating an example of the movement trajectory generation processing according to the present embodiment.
FIG. 6A shows position coordinates (first position information) based on the detection information detected by the laser radar device 20. In addition, FIG. 6B illustrates position coordinates (second position information) based on image information captured by the image capturing device 30. Here, the position coordinates (first position information) shown in FIG. 6A are shown by black circles, and the position coordinates (second position information) shown in FIG. 6B are shown by white circles. Times t1 to t21 indicate time information when each position coordinate is detected. In addition, in this example, in order to simplify the description, the position coordinates are described as two-dimensional coordinates on the XY axes.

The laser radar device 20 can measure accurate position information without imposing a processing load, but since the sampling frequency is low, the position coordinates (first position information) shown in FIG. 6A are intermittently detected. become. Further, since the image information can be continuously captured, the position coordinates (second position information) shown in FIG. 6B are continuously detected. However, the position coordinate (second position information) has a large processing load when detecting the position coordinate from the image information, and its accuracy is inferior to that of the position coordinate (first position information).

Further, FIG. 6C shows a state in which the position coordinates (first position information) shown in FIG. 6A and the position coordinates (second position information) shown in FIG. 6B are superimposed. .. Further, FIG. 6D shows a locus in which the position coordinates (first position information) shown in FIG. 6A and the position coordinates (second position information) shown in FIG. 6B are mutually complemented. ing. That is, the trajectory generation unit 124 mutually complements the position coordinates (first position information) based on the detection information and the position coordinates (second position information) based on the image information to generate the position coordinates of the target object. Finally, the movement locus as shown by the movement locus L0 in FIG. 6D is generated. The locus generation unit 124 stores the generated movement locus in the movement locus storage unit 114 in association with the identification information of the object, the time information, and the three-dimensional position coordinates, as shown in FIG. 4, for example. The identification information here is an identification ID given to the detected object by a predetermined rule.
The laser radar device 20 and the imaging device 30 are arranged so that the three-dimensional coordinates based on the detection information in the target area and the three-dimensional coordinates based on the image information match.

Further, the trajectory generation unit 124 corrects the position information on the plane of the target area based on the first position information based on the detection information, and the position in the height direction of the target area based on the second position information based on the image information. The information may be corrected to generate three-dimensional coordinates (three-dimensional position information) including position information on the plane and position information in the height direction. Here, an example of the three-dimensional coordinate calculation process according to the present embodiment will be described with reference to FIG. 7.

FIG. 7 is a diagram illustrating an example of a three-dimensional coordinate calculation process according to this embodiment.
In FIG. 7, an image G1 indicates image information captured by the image capturing device 30. Here, it is assumed that the court AR2 is a volleyball court and the player PL2 is about to hit the ball B1. In such a case, it is difficult to determine from the image G1 whether or not the player PL2 is jumping.

Therefore, in order to generate accurate three-dimensional coordinates of the player PL2, the trajectory generation unit 124, based on the detection information detected by the laser radar device 20, coordinates of the position PT1 on the XY plane on the court AR2 of the player PL2 ( x1, y1) is calculated. Then, the trajectory generation unit 124 calculates the position coordinate (z1) in the height direction (Z-axis direction) of the player PL2 from the position of the player PL2 on the image information (image G1) imaged by the imaging device 30, and the player. Three-dimensional coordinates (x1, y1, z1) of PL2 are generated.

The locus generation unit 124 calculates the position coordinates (z1) of the player PL2, and from the positional relationship on the screen between the player PL2 and the net NT, the poles (P1, P2), the line LN of the court AR2, and the like. The position coordinate (z1) of the player PL2 is calculated. Further, the laser radar device 20 and the image pickup device 30 are configured to detect three-dimensional coordinates based on the detection information and three-dimensional coordinates based on the image information from the positional relationship between the net NT, the poles (P1, P2) and the line LN in the court AR2. It is arranged so that it matches the dimensional coordinates. In addition, the trajectory generation unit 124 may correct the position information on the plane of the target area based on the first position information and the second position information in the above-described processing.

Further, the trajectory generation unit 124 generates the movement trajectory of the target object by associating the position information of the target object in the generated target area with the identification information provided by the identification processing unit 123. That is, the locus generation unit 124 reads the movement locus of the object stored in the movement locus storage unit 114, associates the movement locus of the object with the identification information of the object provided by the identification processing unit 123, and associates the object with the target locus. It is stored in the object trajectory storage unit 115 as the movement trajectory of the. Specifically, the trajectory generation unit 124, for example, as illustrated in FIG. 5, identification information of an object (“object name”), time information (“time”), and three-dimensional position coordinates (“coordinates”). ) Are associated with each other and stored in the movement locus storage unit 114.

When identifying the target person as the target object, the identification processing unit 123 identifies the target person based on image recognition of the face or clothing of the target person. That is, the identification processing unit 123 requests the image processing unit 122 for identification, and the image processing unit 122 identifies the target person based on the feature amount of the face or clothing of the target person stored in the target object information storage unit 113. To do. The identification processing unit 123 identifies the target person based on the result of image recognition by the image processing unit 122.
Here, the identification process of the target person by the identification processing unit 123 will be described with reference to FIG. 8.

FIG. 8 is a diagram illustrating an example of target person identification processing according to the present embodiment.
A process of recognizing a target person when the player PL3 (player B) and the player PL4 (player A) run across the court AR3 in FIG. 8 will be described.

As shown in FIG. 8A, when the player PL3 (player B) and the player PL4 (player A) cross the court AR3, the trajectory generation unit 124 identifies the object detected by the laser radar device 20. IDs (for example, ID1 to ID4) are given, and a movement locus of each object is generated as shown in FIG. 8B. In FIG. 8B, the horizontal axis represents time and the vertical axis represents the identification ID of each object.

Further, the circles (circles) and the crosses (crosses) in the graph indicate the times when the identification processing unit 123 requests the image processing unit 122 to identify the object. Further, a circle (circle) indicates that the object (player PL3 or player PL4) to which the identification ID is assigned can be identified by the image processing unit 122, and a cross (cross) indicates the identification ID. It indicates that the identification of the object has failed. When the identification processing unit 123 fails to identify the object to which the identification ID is added, after a certain interval, the identification processing unit 123 requests the image processing unit 122 to identify the object again. No request is made to identify the object with the identification ID. As a result, the identification processing unit 123 reduces the processing load for identifying the object. The locus generation unit 124 stores the generated movement locus in the movement locus storage unit 114 as illustrated in FIG. 4 described above.

Further, the identification processing unit 123, based on the partial image of the position at which the object (player PL3 or player PL4) is detected and the feature amount stored in the object information storage unit 113 for each movement trajectory, The player PL3 or player PL4) is identified and the object name (player PL3 or player PL4) is given. For example, as illustrated in 8(c), the identification processing unit 123 assigns “player B” to the player PL3 and “player A” to the player PL4. As shown in FIG. 8D, the trajectory generation unit 124 associates the object name given by the identification processing unit 123 with the movement trajectory stored in the movement trajectory storage unit 114.

In FIGS. 8C and 8D, the movement track of "player A" is the movement track L1, and the movement track of "player B" is the movement track L2. The locus generation unit 124 stores a movement locus with object names such as “player A” and “player B” stored in the object locus storage unit 115, as shown in FIG. 5, for example.

Next, the operation of the position tracking system 1 according to the present embodiment will be described with reference to the drawings.
FIG. 9 is a flowchart showing an example of the operation of the position tracking system 1 according to this embodiment. In this figure, the laser radar device 20 constantly detects the detection information, and the imaging device 30 constantly picks up the image information.

In FIG. 9, the information processing device 10 of the position tracking system 1 first acquires and synchronizes the detection information of the laser radar device 20 and the image information (step S101). The data acquisition unit 121 of the information processing device 10 acquires the detection information from the laser radar device 20 and the image information from the imaging device 30. The data acquisition unit 121 stores the detection information acquired from the laser radar device 20 and the time information in the radar detection storage unit 111 in association with each other. Further, the data acquisition unit 121 stores the image information acquired from the imaging device 30 and the time information in the image information storage unit 112 in association with each other.

Next, the information processing device 10 generates position information in the target area based on the detection information of the laser data device and the image information (step S102). The locus generation unit 124 of the information processing device 10 uses the detection information (for example, the distance and direction to the object) detected by the laser radar device 20 to detect the position information (first position) of the detected object (target object) in the target area. Position information) is generated. The image processing unit 122 also generates position information (second position information) of the object (target object) in the target area based on the image information, for example. Here, the first position information and the second position information are, for example, three-dimensional coordinates of the center of gravity of the object. The locus generation unit 124 mutually complements the first position information and the second position information to correct and generate the three-dimensional coordinates of the object.

Next, the information processing device 10 cuts out a partial image corresponding to the position information, performs image recognition on the partial image, and adds identification information (step S103). The identification processing unit 123 of the information processing device 10 causes the image processing unit 122 to extract the partial image in the image information corresponding to the position of the object of the first position information described above. The identification processing unit 123 causes the image processing unit 122 to identify the target object based on the extracted partial image and the feature amount stored in the target object information storage unit 113, and the target object is identified with respect to the identified target object. Identification information for identifying an object is added.

Next, the information processing device 10 generates a movement locus of the object to which the identification information is added, based on the position information (step S104). The trajectory generation unit 124 associates the generated three-dimensional coordinates of the target object, the identification information provided by the identification processing unit 123, and the time information to generate a movement trajectory of the target object. For example, as shown in FIG. 5, the trajectory generation unit 124 associates the three-dimensional coordinates of the object, the identification information of the object, and the time information with each other, and defines the movement trajectory storage unit 114 as the movement trajectory of the object. To memorize. After the processing of step S104, the trajectory generation unit 124 ends the processing.

As described above, the position tracking system 1 (position measurement system) according to the present embodiment includes the laser radar device 20 (detection device), the imaging device 30, the data acquisition unit 121 (acquisition unit), and the trajectory generation unit 124. (Position measuring unit). The laser radar device 20 irradiates a target area with laser light, and detects detection information including at least the distance and direction to the object based on the reflected light from the object. The imaging device 30 captures an image including the target area. The data acquisition unit 121 acquires and synchronizes the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. The trajectory generation unit 124 includes the first position information, which is the position information of the object in the target area based on the detection information acquired by the data acquisition unit 121, and the position information of the object in the target area based on the image information acquired by the data acquisition unit 121. And position information of the target object is generated by complementing the second position information. The trajectory generation unit 124 generates a movement trajectory of the object on which the object has moved based on the generated position information of the object.

As a result, the position tracking system 1 according to the present embodiment can measure accurate position information without imposing a processing load, but the laser radar device 20 that is intermittently detected due to the low sampling frequency and the processing load. In addition to being large, the accuracy of the position information is inferior, but by using together with the imaging device 30 capable of continuously capturing images, it is possible to measure the position of the object with high accuracy while suppressing the processing load. The position tracking system 1 according to the present embodiment can realize accurate three-dimensional position coordinate measurement, which is difficult only with the image information of the imaging device 30, by using the laser radar device 20 together. Therefore, the position tracking system 1 according to the present embodiment can track the position of the object with high accuracy while suppressing the processing load.

Further, in the present embodiment, the trajectory generation unit 124 mutually complements the first position information and the second position information, corrects the position information of the target object, and based on the corrected position information of the target object. , Generate a movement trajectory of the object.
As a result, the position tracking system 1 according to the present embodiment mutually complements the first position information based on the detection information of the laser radar device 20 and the second position information based on the image information of the imaging device 30, for example, Therefore, it is possible to perform the robust and highly accurate position measurement of the object without being affected by the complicated background or the external environment. For example, as shown in FIG. 8, the position tracking system 1 according to the present embodiment can accurately generate the movement trajectory even when the objects overlap each other.

In addition, the position tracking system 1 according to the present embodiment complements each other to detect position information with high accuracy, so that it is not necessary to increase the number of imaging devices 30 that have a large processing load for image processing. For example, real-time processing can be performed while suppressing the processing load. Further, in the position tracking system 1 according to the present embodiment, for example, it is not necessary to attach an accessory such as a marker to an object, and the system configuration can be a system (simplified).

Further, in the present embodiment, the trajectory generation unit 124 corrects the position information on the plane of the target area based on the first position information, and corrects the position information in the height direction of the target area based on the second position information. Then, the three-dimensional position information including the position information on the plane and the position information in the height direction is generated, and the trajectory of the object is generated based on the generated three-dimensional position information.
As a result, the position tracking system 1 according to the present embodiment can measure an accurate position in the height direction (Z-axis direction), which is difficult to measure from image information from image information, as shown in FIG. The radar device 20 can measure an accurate position in the depth direction (XY axis direction). Therefore, the position tracking system 1 according to the present embodiment can generate the trajectory of the target object with higher accuracy.

In addition, in the present embodiment, the laser radar device 20 and the imaging device 30 are arranged such that the first position information and the second position information in the target area match.
Thereby, the position tracking system 1 according to the present embodiment can measure the position of the target object with higher accuracy.

The position tracking method (position measuring method) according to the present embodiment is a position tracking method of the position tracking system 1 (position measuring system) including the laser radar device 20 (detecting device) and the imaging device 30, and is acquired. And a trajectory generation step (position measurement step). In the acquisition step, the data acquisition unit 121 acquires and synchronizes the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. In the trajectory generation step (position measurement step), the trajectory generation unit 124 (position measurement unit) acquires the first position information, which is the position information of the object in the target area based on the detection information acquired in the acquisition step, and the acquisition step. The position information of the target object is generated by complementing the second position information that is the position information of the object in the target area based on the generated image information, and the target object moves based on the generated position information of the target object. Generate a movement trajectory of the object.
As a result, the position tracking method according to the present embodiment has the same effects as the position tracking system 1 described above, and it is possible to measure the position of the object with high accuracy while suppressing the processing load.

The position tracking system 1 according to the present embodiment also includes a laser radar device 20, an imaging device 30, a data acquisition unit 121, an identification processing unit 123, and a trajectory generation unit 124. The data acquisition unit 121 acquires and synchronizes the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. The identification processing unit 123 uses the image processing unit 122 to target the object based on the partial image in the image information corresponding to the position of the object detected based on the detection information acquired by the data acquisition unit 121. The identification information for identifying the target object is added to the identified target object. The trajectory generation unit 124 associates the position information of the target object in the target area generated based on the detection information and the image information with the identification information provided by the identification processing unit 123, and generates the movement trajectory of the target object. To do.

As a result, the position tracking system 1 according to the present embodiment can identify the object with high accuracy by using the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. it can. The position tracking system 1 according to the present embodiment performs identification processing on a partial image corresponding to the position of a detected object, for example, and therefore, compared to the case of performing identification processing on the entire image information, the background It is possible to significantly reduce the processing load as well as to reduce noise such as. Therefore, the position tracking system 1 according to the present embodiment can accurately track the position of the measurement target while suppressing the processing load.

Further, in the present embodiment, the detection information of the laser radar device 20 includes the contour of the object. The identification processing unit 123 uses the image processing unit 122 to extract a partial image in the image information based on the contour of the object detected by the laser radar device 20, and based on the extracted partial image, target the object to the target object. Identify as.
Thereby, the position tracking system 1 according to the present embodiment can further remove noise such as background by cutting out a partial image from the image information along the contour of the object, and recognize the object by image processing. The accuracy can be further improved.

Further, in the present embodiment, the target includes a target person. When identifying the target person as the target object, the identification processing unit 123 uses the image processing unit 122 to identify the target person based on image recognition of the face or clothing of the target person.
Thereby, the position tracking system 1 according to the present embodiment can appropriately identify a target person such as a player or a player.

In addition, in the present embodiment, the trajectory generation unit 124 has the first position information, which is the position information of the object in the target area based on the detection information, and the second position information, which is the position information of the target object in the target area based on the image information. Are mutually complemented to correct the position information of the target object, and the corrected position information of the target object and the identification information provided by the identification processing unit 123 are associated with each other to generate a movement trajectory of the target object.
As a result, the position tracking system 1 according to the present embodiment can track the position of the measurement target object with higher accuracy.

Further, the position tracking method according to the present embodiment irradiates a target area with laser light, and based on the reflected light from the object, the position tracking method of the position tracking system 1 including the laser radar device 20 and the imaging device 30. That is, it includes an acquisition step, an identification processing step, and a trajectory generation step. In the acquisition step, the data acquisition unit 121 acquires and synchronizes the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. In the identification processing step, the identification processing unit 123 uses the image processing unit 122 to correspond to the position of the object detected based on the detection information acquired in the acquisition step, based on the partial image in the image information, An object is identified as an object, and identification information for identifying the object is added to the identified object. In the trajectory generation step, the trajectory generation unit 124 associates the position information of the target object in the target area generated based on the detection information and the image information with the identification information provided by the identification processing step to associate the target object. Generates the movement locus of.
As a result, the position tracking method according to the present embodiment has the same effects as the position tracking system 1, and it is possible to accurately track the position of the measurement target while suppressing the processing load.

[Second Embodiment]
Next, a play determination system 1a according to the second embodiment will be described with reference to the drawings.
In the present embodiment, a play determination system 1a that includes the position tracking system 1 described above and determines the play of the target person that indicates the motion and behavior of the target person based on the movement trajectory of the target person and the skeleton and joints of the target person. Will be described.

FIG. 10 is a block diagram showing an example of the play determination system 1a according to the present embodiment.
As shown in FIG. 10, the play determination system 1a (an example of a determination system) includes an information processing device 10a, a laser radar device 20, and an imaging device 30.
In this figure, the same components as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted. Further, in the present embodiment, the laser radar device 20 and the imaging device 30 are an example of a detection device capable of detecting at least the shape and position information of the object.

The information processing device 10a is, for example, a computer device such as a PC. The information processing device 10a, like the information processing device 10 of the first embodiment described above, targets in the target area based on the detection information detected by the laser radar device 20 and the image information captured by the imaging device 30. The position information of the object is generated, and the movement locus of the object is generated based on the generated position information. Further, the information processing device 10a estimates the skeleton and joints of the target person from the image information, and determines the play of the target person from the skeletons and joints of the target person and the movement trajectory of the target person. Here, the play means, for example, a motion and a behavior of a target person in sports.
Further, the information processing device 10a includes a storage unit 11a and a control unit 12a.

The storage unit 11a stores information used for various processes of the information processing device 10a. The storage unit 11a includes a radar detection storage unit 111, an image information storage unit 112, an object information storage unit 113, a movement trajectory storage unit 114, an object trajectory storage unit 115, a skeletal joint information storage unit 116, and The determining device information storage unit 117 and the determination result storage unit 118 are provided.

The skeletal joint information storage unit 116 stores skeletal joint information estimated by the skeleton estimating unit 125 described later. The skeletal joint information storage unit 116 stores, for example, the identification information of the target person, the time information, and the skeletal joint information indicating the skeleton and the joint in association with each other. Here, the skeletal joint information is, for example, three-dimensional position coordinates of the skeleton and the joint.

The determiner information storage unit 117 stores information on a determiner for determining play. The determiner information storage unit 117 is machine-learned so as to determine the play of the target person, which indicates the motion and behavior of the target person, based on the past trajectory of the target person and the skeleton and joints of the target person, for example. The result and the like are stored as information of the determiner. Here, the machine learning includes, for example, decision tree learning, neural networks, support vector machines (SVM), various regression models, deep learning (deep learning), and the like. Further, the information of the determiner includes various weighting information, rule information, various parameters, and the like.
In addition to the type of play, the play determination includes success/failure of the play, performance, score determination, and the like.

The determination result storage unit 118 stores the determination result determined by the play determination unit 126 described later. The determination result storage unit 118 stores, for example, the identification information of the target person, the time information, and the play determination result in association with each other.

The control unit 12a is, for example, a processor including a CPU and controls the information processing apparatus 10a in a centralized manner. The control unit 12a executes the same process as the control unit 12 of the first embodiment described above, and determines, for example, the play of the target person.
The control unit 12a includes a data acquisition unit 121, an image processing unit 122, an identification processing unit 123, a trajectory generation unit 124, a skeleton estimation unit 125, a play determination unit 126, and a play evaluation unit 127.

The skeleton estimation unit 125 estimates the skeleton and joints of the target person based on the detection information detected by the detection device (for example, the laser radar device 20 and the imaging device 30). The skeleton estimation unit 125 estimates the skeleton and joints of the target person based on the contour of the target person detected by the laser radar device 20 and the image information including the target person imaged by the imaging device 30. The skeleton estimation unit 125 stores the estimated skeleton and skeletal joint information in association with the skeletal joint information, the identification information of the target person, and the time information in the skeletal joint information storage unit 116. The skeleton estimating unit 125 estimates the skeleton and joints of the target person using, for example, an existing skeleton tracking technique. The skeleton estimating unit 125 may estimate the line of sight of the target person in addition to the skeleton and joints. In this case, the skeletal joint information may include information indicating the line of sight. Further, the skeleton estimating unit 125 may estimate one of the skeleton and the joint of the target person.

The image processing unit 122 and the identification processing unit 123 are the same as those in the first embodiment, and therefore their description is omitted here. If the target person cannot be identified based on the image recognition of the face or clothing of the target person, the identification processing unit 123 determines the skeleton or joint of the target person (skeleton and joint of the target person, or One of the skeleton and the joint of the target person) may be estimated, and the target person may be identified based on the estimated skeleton or joint and a predetermined feature of the skeleton or joint of the target person. That is, the target object information storage unit 113 stores, for example, the feature amount of the skeleton or joint of the target person in advance, and the identification processing unit 123 determines the skeleton or joint estimated by the skeleton estimating unit 125 described above and the target object. The target person may be identified based on the feature amount of the skeleton or joint of the target person stored in the information storage unit 113.

The locus generation unit 124 is the same as that in the first embodiment, and therefore its description is omitted here. That is, the trajectory generation unit 124 generates the movement trajectory of the target person based on the detection information detected by the detection device (for example, the laser radar device 20 and the imaging device 30).
In addition to the movement trajectory of the target person, the trajectory generation unit 124 in the present embodiment also includes movement trajectories of objects around the target person, movement trajectories of skeletons and joints, and parts of the body such as the hands and feet of the target person. May be generated.

The play determination unit 126 is based on the skeleton and joints of the target person estimated by the skeleton estimation unit 125, the movement trajectory of the target person generated by the trajectory generation unit 124, and a determiner that determines the play of the target person. Determine the play of a person. Here, the determiner is generated so as to determine the play of the target person, which indicates the motion and behavior of the target person, based on the past movement trajectory of the target person and the skeleton and joints of the target person. It is generated by learning. The play determination unit 126 uses the determiner based on the determiner information stored in the determiner information storage unit 117 to determine the play of the target person from the skeleton and joints of the target person and the movement trajectory of the target person.

Here, the play of the target person is, for example, a player's receive, attack, serve, block, etc. in volleyball or screen play in basketball. In addition, for example, when determining a play in volleyball, the play determination unit 126 exists in the vicinity of the target person's skeleton and joints and the movement path of the object such as a ball, in addition to the movement trajectory of the target person. The play of the target person may be determined based on the movement trajectory of the target object. In this case, the trajectory generation unit 124 generates a movement trajectory of an object (for example, a ball) existing around the target person, and the play determination unit 126 uses the determination device to determine the movement trajectory and skeleton of the target person. Also, the play of the target person is determined from the joint and the movement trajectory of the target (for example, the ball).

In addition, for example, when determining screen play in basketball, the play determination unit 126 may determine the play of the target person based on the movement trajectory of the person around the target person. In this case, the trajectory generation unit 124 generates a movement trajectory of an object (for example, a person in the vicinity) existing around the target person, and the play determination unit 126 uses the determination device to determine the movement trajectory of the target person. , The play of the target person is determined from the skeleton and joints and the movement trajectory of the target object (for example, a person in the vicinity).
The play determination unit 126 stores the information indicating the determined play as a determination result in the determination result storage unit 118 in association with the target person and the time information.

The play evaluation unit 127 evaluates the play based on the play determination result determined by the play determination unit 126. For example, the play evaluation unit 127 totals the number of times of execution of a play such as a jump during training from the play determination result, and evaluates the effect of training, efficiency, overtraining, and the like. In addition, the play evaluation unit 127 totalizes the plays executed in the match based on the play determination result, and evaluates the match content based on the total result of the play.

Next, the operation of the play determination system 1a according to the present embodiment will be described with reference to the drawings.
FIG. 11 is a flowchart showing an example of the operation of the play determination system 1a according to this embodiment. In this figure, the laser radar device 20 constantly detects the detection information, and the imaging device 30 constantly picks up the image information.
Since the processing from step S201 to step S203 shown in FIG. 11 is the same as the processing from step S101 to step S103 shown in FIG. 9, the description thereof is omitted here.

In step S204, the information processing device 10a estimates the skeleton and the joint based on the detection information and the image information. The skeleton estimating unit 125 of the information processing device 10a determines the skeleton and joint of the target person based on, for example, the contour of the target person detected by the laser radar device 20 and the image information including the target person captured by the imaging device 30. presume. The skeleton estimation unit 125 stores the estimated skeleton and skeletal joint information in association with the skeletal joint information, the identification information of the target person, and the time information in the skeletal joint information storage unit 116.

Next, the information processing device 10a generates a movement trajectory of the measurement target to which the identification information is added, based on the position information (step S205). The locus generation unit 124 of the information processing device 10a associates the generated three-dimensional coordinates of the target object, the identification information provided by the identification processing unit 123, and the time information with each other, as in step S104 illustrated in FIG. Generate a movement trajectory of the object.

Next, the information processing device 10a makes a play determination based on the feature amount of the movement trajectory and the skeleton and joint (step S206). The play determination unit 126 of the information processing device 10a uses the determiner based on the determiner information stored in the determiner information storage unit 117 to determine, from the skeleton and the joint of the target person and the feature amount of the movement trajectory of the target person, Determine the play of the target person. The play determination unit 126 stores the determination result of the play in the determination result storage unit 118.

Next, the information processing device 10a determines whether or not the play determination is finished (step S207). When the play determination is finished (step S207: YES), the information processing device 10a advances the process to step S208. When the play determination is not finished (step S207: NO), the information processing apparatus 10a returns the process to step S201 and repeats the processes from step S201 to step S207.

In step S208, the information processing device 10a evaluates the play determination result. The play evaluation unit 127 of the information processing device 10a evaluates the play based on the play determination result determined by the play determination unit 126. The play determination unit 126 ends the process after the process of step S208.
In the example described above, the skeleton estimation unit 125 estimates the skeleton and joints of the target person, and the play determination unit 126 determines the play based on the movement trajectory of the target person and the positions of the skeletons and joints of the target person. However, the skeleton estimating unit 125 estimates one of the skeleton and joints of the target person, and determines the play based on the movement trajectory of the target person and one of the positions of the skeleton and joints of the target person. You may do so.

As described above, the play determination system 1a (determination system) according to the present embodiment includes the detection device (for example, the laser radar device 20 and the imaging device 30), the skeleton estimation unit 125, the trajectory generation unit 124, and the play determination. And a section 126. The detection device (for example, the laser radar device 20 and the imaging device 30) can detect at least the shape and position information of the object. The skeleton estimation unit 125 estimates the skeleton or joint of the target person based on the detection information detected by the detection device. The trajectory generation unit 124 generates a trajectory of the target person based on the detection information detected by the detection device. The play determination unit 126 determines the play of the target person based on the determiner, the skeleton or joint of the target person estimated by the skeleton estimation unit 125, and the movement trajectory of the target person generated by the trajectory generation unit 124. Here, the determiner is generated to determine the play of the target person, which indicates the motion and behavior of the target person, based on the past trajectory of the target person and the skeleton or joint of the target person.

As a result, the play determination system 1a according to the present embodiment determines the play based on the skeleton or joint of the target person and the movement trajectory of the target person, so that the processing load is suppressed and the play of the target person is performed with high accuracy. A decision can be made. Further, in the play determination system 1a according to the present embodiment, for example, the player's play, position, time, success or failure of play, performance, score determination, etc. can be automatically recorded without burdening the user. That is, the play determination system 1a according to the present embodiment can automatically record the score of the sports competition.

In addition, in the present embodiment, the trajectory generation unit 124 generates the movement trajectory of the target object existing around the target person based on the detection information detected by the detection device (for example, the laser radar device 20 and the imaging device 30). To do. The play determination unit 126 determines the play of the target person from the movement trajectory of the target person, the skeleton or joint of the target person, and the movement trajectory of the target person using the determiner.
As a result, the play determination system 1a according to the present embodiment can improve the precision of the play determination by using the movement trajectory of the object, which is a tool such as a ball or a racket.

For example, in a volleyball, the play determination system 1a according to the present embodiment determines, for example, a receive, an attack, a serve, a block, etc. by using the movement trajectory of an object such as a ball as a determiner to determine the play. be able to.
Further, for example, in the basket, the play determination system 1a according to the present embodiment accurately determines screen play, for example, by using the movement trajectory of the person around the target person as a determiner to determine the play. be able to.
Further, for example, in tennis, by using the skeleton of the target person and the movement trajectory of the target person, the ball, and the racket as a determiner, the play determination is performed to determine the tennis shot type (for example, backhand, forehand, smash). , A slice, etc.), a ball (eg, speed, spin type, number of revolutions, etc.), success or failure of a shot, and the like.

Further, in the present embodiment, the detection device includes the laser radar device 20 and the imaging device 30. The skeleton estimation unit 125 estimates the skeleton or joint of the target person based on the contour of the target person detected by the laser radar device 20 and the image information including the target person imaged by the imaging device 30. The trajectory generation unit 124 generates first position information that is position information of the target person in the target area based on detection by the laser radar device 20, and second position information that is position information of the target person in the target area based on image information. Complementing each other, the position information of the target person is corrected, and the movement trajectory of the target person is generated based on the corrected position information of the target person.

Thereby, the play determination system 1a according to the present embodiment complements the first position information based on the detection information of the laser radar device 20 and the second position information based on the image information of the imaging device 30 to perform processing. The position of the object can be measured while suppressing the load. Therefore, the play determination system 1a according to the present embodiment can perform the play determination of the target person with higher accuracy.

Further, the play determination system 1a according to the present embodiment identifies the target person based on the partial image in the image information corresponding to the first position information based on the detection information of the laser radar device 20, and identifies the identified target person. An identification processing unit 123 for providing identification information for identifying the target person. The locus generation unit 124 generates the movement locus of the target person in association with the identification information provided by the identification processing unit 123.
As a result, the play determination system 1a according to the present embodiment can accurately identify an object by using the detection information detected by the laser radar device 20 and the image information representing the image captured by the imaging device 30. Therefore, the position of the measurement target can be tracked with high accuracy while suppressing the processing load. Therefore, the play determination system 1a according to the present embodiment can perform the play determination of the target person with higher accuracy.

In the play determination system 1a according to the present embodiment, the image pickup device 30 can detect the position information with high accuracy by using the position information that is difficult to obtain with high accuracy together with the laser radar device 20. .. Therefore, the play determination system 1a according to the present embodiment can broaden the types of play that can be determined, and can be applied to analysis of various competitions and games. Further, the play determination system 1a according to the present embodiment improves the determination speed and the determination accuracy by being used in combination with the laser radar device 20 in the play determination that requires a complicated and high processing load in the imaging device 30. be able to.

Further, in the present embodiment, when the target person cannot be identified based on the image recognition of the face or clothing of the target person, the identification processing unit 123 estimates and estimates the skeleton or joint of the target person based on the partial image. The target person is identified based on the skeleton or joint and the predetermined characteristics of the skeleton or joint of the target person.
As a result, the play determination system 1a according to the present embodiment can identify the object with higher accuracy.

Further, in the present embodiment, the trajectory generation unit 124 generates position information on the plane of the target area based on the detection information of the laser radar device 20, and the height direction of the target area based on the image information of the imaging device 30. Position information is generated, three-dimensional position information including position information on a plane and position information in the height direction is generated, and a trajectory of an object (target person) is generated based on the generated three-dimensional position information. To do.
As a result, the play determination system 1a according to the present embodiment can measure an accurate position, so that the play determination involving a jump can be performed with high accuracy, and the types of play that can be determined can be further expanded.

Further, the play determination system 1a according to the present embodiment includes a play evaluation unit 127 that evaluates a play.
As a result, the play determination system 1a according to the present embodiment can evaluate, for example, the mileage of each player during the practice, the number of jumps or the height, and the play action, and perform the management such as overtraining in a unified manner. You can

Further, the determination method according to the present embodiment includes a skeleton estimation step, a trajectory generation step, and a play determination step. In the skeleton estimating step, the skeleton estimating unit 125 estimates the skeleton or joint of the target person based on at least the detection information detected by the detection device capable of detecting the shape and position information of the object. In the locus generation step, the locus generation unit 124 generates the movement locus of the target person based on the detection information detected by the detection device. In the play determination step, the play determination unit 126 is generated so as to determine the play of the target person indicating the motion and action of the target person based on the past movement trajectory of the target person and the skeleton or joint of the target person. The play of the target person is determined based on the determiner, the skeleton or joint of the target person estimated by the skeleton estimation unit 125, and the movement trajectory of the target person generated by the trajectory generation unit 124.
As a result, the determination method according to the present embodiment has the same effect as that of the above-described play determination system 1a, and can perform the play determination of the target person with high accuracy while suppressing the processing load.

It should be noted that the present invention is not limited to each of the above-described embodiments, and can be modified within a range not departing from the spirit of the present invention.
For example, the numbers of the laser radar devices 20 and the imaging devices 30 included in the position tracking system 1 (play determination system 1a) are not limited to those in the above embodiments, and other numbers may be included.

In each of the above-described embodiments, the information processing device 10 (10a) has been described as an example of one device. However, the information processing device 10 (10a) is not limited to this, and may be composed of a plurality of devices. Further, in the information processing device 10 (10a), an external device may include part or all of the storage unit 11 (11a). Further, in this case, the external device may be a storage device connectable to the information processing device 10 (10a) via the network. Further, in the information processing device 10 (10a), an external device may include a part of the functions of the control unit 12 (12a).

In each of the above embodiments, a laser radar device is used as an example of a detection device that irradiates a target area with a wave and detects detection information including at least a distance and a direction to the object based on a reflected wave from the object. Although the example using 20 has been described, the present invention is not limited to this. For example, the detection device may irradiate a light wave, a radio wave, or a sound wave to detect detection information including at least a distance and a direction to the object based on a reflected wave from the object. Here, the wave is a wave that propagates in space, such as a light wave such as a laser beam, a radio wave, a sound wave such as an ultrasonic wave, or the like.

In the above-described second embodiment, an example in which the play determination unit 126 determines the play based on the movement trajectory of the target person, the positions of the skeleton and the joint, and the movement trajectory of the target object existing around the target person. However, the present invention is not limited to this. The play determination unit 126 is based on, for example, movement trajectories of skeletons and joints, positional relationships with surrounding objects such as poles, nets, and coat lines, and trajectories of part of the body such as the hands and feet of the target person. Then, the play may be determined. The play determination unit 126 may determine the play based on one of the movement trajectory of the target person and the positions of the skeleton and joints.
The play determined by the play determination unit 126 may be, for example, a soccer goal, a line-in or line-out of tennis or volleyball, the name of a technique such as judo, or the like. Further, the play determination unit 126 may determine, as the play determination, success or failure of the play (for example, basketball success/failure), performance/score (for example, gymnastics score).

Further, in each of the above-described embodiments, an example in which the target area is, for example, a court that is a stadium for playing sports, a game field, or the like has been described, but the present invention is not limited to this. The target area to which the information processing device 10 (10a) is applied may be, for example, an ice rink such as skating, a competition place on snow such as ski jumps and half pipes, and a pool such as a swimming race and artistic swimming. , It may be a place for other competitions.

Further, in each of the above-described embodiments, the laser radar device 20 and the imaging device 30 are arranged so that the first position information and the second position information in the target area match each other. You may go to
For example, in a competition using a court, a calibration process of the laser radar device 20 and the imaging device 30 may be performed at the same time by using a colored sticker made of a retroreflective material on the court line. In this case, the seal or the like may be attached to a tool such as a net pole or a goal pole whose position is fixed during the competition for use.

The position measurement process of the information processing device 10 (10a) is not limited to the above-described embodiments, and may be changed as follows depending on the sporting event or purpose.
For example, the trajectory generation unit 124 of the information processing device 10 (10a) normally performs the position measurement by complementing the first position information by the laser radar device 20 with the second position information based on the image information. When used in a wheelchair basketball or wheelchair rugby competition, it is considered that the player often holds the ball and it is difficult for the laser radar device 20 to detect the first position information. In such a case, the locus generation unit 124 may supplement the second position information based on the image information mainly with the first position information by the laser radar device 20.

Further, as described above, the laser radar device 20 can accurately detect the position in real time, but since the sampling frequency is low, the position information becomes fragmentary. On the other hand, in position detection based on image information, continuous position information can be detected, but the processing load is high and it takes time. Therefore, when placing importance on real-time position detection, the trajectory generation unit 124 mainly reduces the ratio, frequency, etc. of complementing the first position information by the laser radar device 20 with the second position information based on the image information. You can In addition, for example, when importance is attached to the accuracy of position detection, the trajectory generation unit 124 may increase the ratio, frequency, etc. of complementation with the second position information based on the image information.
In this way, the trajectory generation unit 124 may change the complementing method between the first position information and the second position information according to the sporting event or the purpose.

Further, in each of the above-described embodiments, the information processing device 10 (10a) may convert the generated movement trajectory into various lines of sight and directions and display the same on a display unit (not shown) or the like. For example, the locus generation unit 124 may convert the movement locus into the line of sight of a baseball catcher or a soccer keeper, and may display the locus, or a ball or shuttle from the line of sight of a player who is served in volleyball, tennis, badminton, or the like. Alternatively, the movement locus may be converted to the locus and displayed.

Further, in each of the above-described embodiments, the information processing device 10 (10a) can cope with a simple facility when it is desired to acquire the trajectory of a moving object in an indoor competition such as a bicycle track competition or speed skating. .. In addition, when measuring speed on medium and long distances on land, it is difficult to distinguish from the video alone, such as when the players overlap in the vicinity of the corner.In the conventional technology, it is difficult to calibrate because many cameras are required. is there. Even in such a case, the information processing device 10 (10a) can acquire the movement trajectory with simple equipment.

Further, by using the information processing device 10 (10a) of each of the above-described embodiments, the following can be dealt with in the operation of the news camera and the like.
For example, the camera can be operated so that the player or the ball is always followed and photographed. In addition, it is possible to process images so that a ball, shuttle, or the like that moves at a speed that human eyes cannot follow is noticeable.

Further, in each of the above-described embodiments, the information processing device 10 (10a) uses the position information and the movement locus to measure the movement locus and flight distance in a ski jump, land, swimming, speed skating, and the like. The recording measurement may be performed.

Further, in the second embodiment described above, the play determination unit 126 may perform the following processing as play determination from the position information, the movement trajectory, the skeleton/joint movement, and the trajectory of the peripheral area.
For example, the play determination unit 126 may notify a change in formation during set play such as soccer, or may predict the success rate to be hit only by a form such as archery. Further, the play determination unit 126 may estimate the condition and the degree of tiredness only with forms such as archery and weight lifting. Further, the play determination unit 126 may determine the play of a player who does not hold the ball, such as basket screen play and volleyball block play. And may be automatically measured.

Further, in the above-described second embodiment, the play evaluation unit 127 may predict the success rate of the technique or the competition based on the play determination result and the past movement trajectory. The play evaluation unit 127 may predict the success rate of, for example, a jump of figure skating or a technique of gymnastics. Further, the play evaluation unit 127 may predict the success rate from the trajectory of the barbell and the movement of the skeleton/joint in weight lifting, for example. This allows the form to be properly managed.

Further, in the above-described second embodiment, the play evaluation unit 127 may visualize the position of the target object, the trajectories of various identified target objects, and the evaluation data. The play evaluation unit 127 may display a display screen (image G2) as shown in FIG.

FIG. 12 is a diagram showing an example of a display screen in a modified example of the play determination system 1a according to the second embodiment.
As a visualization application, the play evaluation unit 127 causes the display unit (not shown) to display various information as shown below, as shown in the image G2.
The play evaluation unit 127 displays, as a visualization application, one image or a time range within the image, for example, the image of the part, the position data, and the accompanying evaluation data. In the image G2, the video, position data, and evaluation data can be displayed/hidden.

Further, the play evaluation unit 127 displays the movement of the position of the equipment such as the player, the ball, and the racket in the target area as the position data in the visualization application, and the position data includes the identified identification information (for example, the identification ID). ) Is displayed. Further, the play evaluation unit 127 has, as the function of the visualization application, for example, the video and the position data are displayed in time synchronization, and the tracking data (tracked movement track) for each player is displayed as the evaluation data. Perform processing such as displaying statistical data such as the number of times, height, maximum speed, and number of sprints.

In addition, as a function of the visualization application, the play evaluation unit 127, for example, in the case of a speed curve, displays video and position data and point data in the graph in association with each other. Processes such as displaying statistical data, displaying a table by a table in addition to displaying statistics such as graphs are executed. As a function of the visualization application, the play evaluation unit 127 may display data of equipment such as a ball and a racket in the evaluation data. For example, the speed of the ball, the number of revolutions, the spin type, and the racket. The speed, shot type, impact position, shot power data, etc. may be displayed.

Next, with reference to FIG. 13, another modification of visualizing an image and evaluation data in the play determination system 1a will be described.
FIG. 13 is a diagram showing another modification of the play determination system 1a according to the second embodiment.

In this figure, the play determination system 1a includes a screen 40 (an example of a display unit) installed above the court AR4 in the valley. The information processing device 10a displays the image captured by the image capturing device 30 on the screen 40 and displays the evaluation data associated with the play in an overlapping manner (image G3 in FIG. 13). For example, in the case of volleyball, the information processing apparatus 10a displays the hitting height of the attack, the ball speed, the ball angle, etc., as shown in the image G3. Further, the information processing device 10a is assumed to allow the user to specify the content and display position of the evaluation data to be superimposed on the video. Further, when the measurement is performed using a plurality of image pickup devices 30, the information processing device 10a can specify which image pickup device 30 the image of the image pickup device 30 is to be displayed on the screen 40. Further, although an example in which the screen 40 is used will be described as an example of the display unit, another display, screen, display device, or the like may be used as long as it is a display device installed around the coat AR4.

Since the display of the evaluation data requires measurement and determination processing, the image displayed on the screen 40 may be reproduced with a delay from real time. In that case, the information processing apparatus 10a may allow the user to set a delay time for displaying the evaluation data and a time for maintaining the display of the evaluation data. In this case, for example, when the delay time is set to 5 seconds in the attack practice, and a plurality of players attack one after another, it is possible to check the current evaluation data and form on the screen 40 after the attack by oneself. it can.
Further, for example, when performing formation practice, the delay time is set to 20 seconds, and a series of plays of receive, toss, and attack can be confirmed on the display such as the screen 40 after the play.

In addition, in the second embodiment, the information processing device 10a outputs the evaluation data generated and accumulated by the play evaluation unit 127 to the outside by text data, a CSV file, or the like according to the user's designation. You can In this case, the evaluation data is, for example, the transition of the hitting point height in one game, the number of blocks and the success rate at the time of the attack, the number of jumps and the height during practice in one day, and the like. In the play determination system 1a, by outputting these evaluation data to the outside, the analyst and the coach can efficiently perform the game analysis and the training evaluation from the output data.

Each of the components included in the position tracking system 1 and the play determination system 1a described above has a computer system inside. Then, a program for realizing the functions of the components of the position tracking system 1 and the play determination system 1a described above is recorded in a computer-readable recording medium, and the program recorded in this recording medium is read into the computer system. By executing the above, the processing in each configuration included in the position tracking system 1 and the play determination system 1a described above may be performed. Here, “reading and executing a program recorded in a recording medium on a computer system” includes installing the program in the computer system. The “computer system” mentioned here includes an OS and hardware such as peripheral devices.
Further, the “computer system” may include a plurality of computer devices connected via a network including a communication line such as the Internet, WAN, LAN, and a dedicated line. Further, the “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in a computer system. As described above, the recording medium storing the program may be a non-transitory recording medium such as a CD-ROM.

The recording medium also includes a recording medium provided inside or outside accessible from the distribution server for distributing the program. It should be noted that the programs are divided into a plurality of programs and downloaded at different timings, and then combined by the respective configurations of the position tracking system 1 and the play determination system 1a, and the distribution server that distributes each of the divided programs is different. May be. Further, the "computer-readable recording medium" holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that serves as a server or a client when the program is transmitted via a network. It also includes things. Further, the program may be for realizing a part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Further, some or all of the above-described functions may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each of the functions described above may be individually implemented as a processor, or part or all of the functions may be integrated and implemented as a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. In addition, when a technology for forming an integrated circuit that replaces the LSI appears due to the progress of semiconductor technology, an integrated circuit according to the technology may be used.

DESCRIPTION OF SYMBOLS 1... Position tracking system, 1a... Play determination system, 10, 10a... Information processing device, 11, 11a... Storage part, 12, 12a... Control part, 20... Laser radar device, 30... Imaging device, 40... Screen, 111 ... radar detection storage unit 112, image information storage unit 113, target object information storage unit 114, movement locus storage unit 115, target locus storage unit 116, skeletal joint information storage unit 117, decision unit information storage Part, 118... Judgment result storage part, 121... Data acquisition part, 122... Image processing part, 123... Identification processing part, 124... Locus generation part, 125... Skeleton estimation part, 126... Play determination part, 127... Play evaluation part

Claims (4)

A detection device capable of detecting at least the shape and position information of the object,
Based on the detection information detected by the detection device, a skeleton estimation unit that estimates the skeleton or joint of the target person,
A trajectory generation unit that generates a movement trajectory of the target person based on the detection information detected by the detection device,
A determiner generated to determine the play of the target person, which indicates the motion and action of the target person, based on the past movement trajectory of the target person and the skeleton or joint of the target person, and the skeleton estimation A skeleton or joint of the target person estimated by the section, and a play determination unit that determines the play of the target person based on the movement trajectory of the target person generated by the trajectory generation unit ,
The detection device includes a laser radar device and an imaging device,
The skeleton estimating unit estimates the skeleton or joint of the target person based on the contour of the target person detected by the laser radar device and image information including the target person imaged by the imaging device,
The trajectory generation unit includes first position information that is position information of the target person in the target area based on detection by the laser radar device, and second position information that is position information of the target person in the target area based on the image information. Information is mutually complemented to correct the position information of the target person, and a movement locus of the target person is generated based on the corrected position information of the target person.
Determination system, wherein a call. The trajectory generation unit generates a movement trajectory of an object existing around the object person based on the detection information,
The play determination unit determines the play of the target person from the movement trajectory of the target person, the skeleton or joint of the target person, and the movement trajectory of the target person using the determiner. The determination system according to claim 1. An identification processing unit that identifies the target person based on the partial image in the image information corresponding to the first position information, and adds identification information for identifying the target person to the identified target person. ,
The determination system according to claim 1 or 2 , wherein the trajectory generation unit generates a movement trajectory of the target person in association with the identification information provided by the identification processing unit. A skeleton estimating step, which estimates a skeleton or joint of the target person, based on detection information detected by a detection device capable of detecting at least the shape and position information of the object,
A trajectory generation unit, a trajectory generation step of generating a movement trajectory of the target person based on the detection information detected by the detection device,
A determiner generated so that the play determination unit determines the play of the target person indicating the motion and action of the target person based on the past movement trajectory of the target person and the skeleton or joint of the target person. And a play determination step of determining the play of the target person based on the skeleton or joint of the target person estimated by the skeleton estimation unit and the movement trajectory of the target person generated by the trajectory generation unit. See
The detection device includes a laser radar device and an imaging device,
In the skeleton estimating step, the skeleton estimating unit, based on the contour of the target person detected by the laser radar device and the image information including the target person captured by the imaging device, the skeleton or joint of the target person. And estimate
In the trajectory generating step, the trajectory generator causes the position information of the target person in the target area based on detection by the laser radar device to be first position information, and the position of the target person in the target area based on the image information. The second position information, which is information, is mutually complemented to correct the position information of the target person, and a movement locus of the target person is generated based on the corrected position information of the target person.
Judgment wherein a call.

Images