KR102544972B1 - Method for tracking sports participants, device for tracking sports participants, and system for tracking sports participants - Google Patents

Abstract

A multi-mode tracking method according to an embodiment of the present application receives a sensor signal from each of a plurality of positioning sensors located of a plurality of SPOSE participants, the sensor signal including a participant identifier (idenfier) and location data. -; receiving captured sports images from cameras positioned around the playfield, the sports images including at least a target participant of a plurality of sports participants within the playfield; detecting, from the sports image, occlusion associated with the target participant; determining a severity of the occlusion based on the sensor signals received from specific positioning sensors mounted on specific sports players located in the region of interest related to the occlusion; if it is not detected that the severity of occlusion is greater than a threshold value, determining a location of the sports participant based on a pixel location associated with the target participant in the sports image; and if it is detected that the severity of the occlusion is greater than a threshold value, determining the location of the sports participant based on the location data of the sensor signal with the participant identifier representing the target participant.

Description

Player tracking method, player tracking device and player tracking system

This application relates to a player tracking method for tracking a player position, a player tracking device, and a player tracking system.

As it becomes possible to analyze the movements and motions of players during sports games due to the improvement of image analysis technology and positioning technology, the field of sports analysis technology that quantifies the performance of sports players is attracting attention. In quantifying the performance of sports players, the analysis of the player's position is considered the most basic element.

Representative methods of calculating the position of a sports player include a method of calculating a point where the athlete is located in a stadium from an image captured by a camera and a method using positioning devices such as GPS or LPS.

However, the positioning method using video has several limitations due to camera technology, including the fact that it cannot respond to occlusion situations such as the player to be tracked in the video being covered by another player, and positioning using a positioning device such as a GPS module. The method also has fundamental problems due to deterioration in positioning accuracy according to the position of the satellite or delay/reflection of signals, and each has limitations in situations where high precision is required.

One object of the present invention is to provide a player tracking method, a player tracking device, and a player tracking system with improved player position tracking accuracy.

A player tracking method disclosed in this application includes receiving a sensor signal from each of a plurality of positioning sensors located in a plurality of sports participants, the sensor signal including a participant identifier (idenfier) and location data; receiving captured sports images from cameras positioned around the playfield, the sports images including at least a target participant of a plurality of sports participants within the playfield; detecting, from the sports image, occlusion associated with the target participant; determining a severity of the occlusion based on the sensor signals received from specific positioning sensors mounted on specific sports players located in the region of interest related to the occlusion; if it is not detected that the severity of occlusion is greater than a threshold value, determining a location of the sports participant based on a pixel location associated with the target participant in the sports image; and if it is detected that the severity of the occlusion is greater than a threshold value, determining the location of the sports participant based on the location data of the sensor signal having the participant identifier representing the target participant.

A player tracking method disclosed herein provides an image-based location representing a first location of a sports participant defined by a first coordinate system, from sports images captured from cameras positioned around a playfield and including the sports participant within the playfield. obtaining data, wherein the image-based location data is obtained based on pixel locations associated with the sports participant within the sports image; detecting an occlusion event related to the sports participant from the sports image; verifying the image-based location data based on a result of the detection; obtaining a positioning sensor-based position representing a second position of the sports participant defined by a second coordinate system based on a sensor signal obtained from a positioning sensor located on the sports participant; obtaining, based on the sensor signal or the second position, a localization sensor-based speed representing the speed of the sports participant defined by the second coordinate system; Preparing a neural network that transforms the position of the second coordinate system into the first coordinate system - The neural network includes an input layer for receiving position and speed according to the second coordinate system, an output layer for outputting a result representing a position value, and the including a hidden layer having a plurality of nodes connecting the input layer and the output layer; preparing a learning set, the learning set including the image-based positioning data determined to be valid and the positioning sensor-based position data and positioning sensor-based velocity data corresponding to the image-based positioning data determined to be valid; and inputting the positioning sensor-based position data and positioning sensor-based speed data of the learning set to the input layer using the learning set, and corresponding to the input positioning sensor-based position data and positioning sensor-based speed data. and learning the neural network by adjusting weights of the nodes based on a difference between the image-based location data and the location value of the result.

A player tracking method disclosed in this application includes receiving sports images captured by cameras located around a playfield, the sports images including sports participants within the playfield; receiving a sensor signal from a positioning sensor mounted on the sports participant; detecting, from the sports image, an occlusion event related to the sports participant; if the occlusion event is not detected, determine a location of the sports participant based on image-based location data representing a location defined in a first coordinate system; When the occlusion event is detected: obtaining a positioning sensor-based position representing a position defined in a second coordinate system based on the sensor signal; obtaining a positioning sensor-based speed indicating a speed defined in the second coordinate system based on the sensor signal or the positioning sensor-based position; and determining a position of the sports participant defined in the first coordinate system from the positioning sensor-based position and the positioning sensor-based velocity using a neural network for converting a position in the second coordinate system to a position in the first coordinate system; The neural network includes an input layer for receiving position and speed according to the second coordinate system, an output layer for outputting a result indicating a position value according to the first coordinate system, and a plurality of layers connecting the input layer and the output layer. It includes a hidden layer having nodes, and can be trained by adjusting weights of the plurality of nodes using positions and velocities according to the second coordinate system labeled with positions according to the first coordinate system.

A player tracking method disclosed in this application includes receiving a sensor signal from a positioning sensor located on a sports participant; obtaining a positioning sensor based position based on the sensor signal; verifying the positioning sensor-based position based on a change of the positioning sensor-based position from a previous positioning sensor-based position; if the positioning sensor-based location is valid, determining the location of the sports participant based on the positioning sensor-based location; If the positioning sensor based location is not valid: obtaining a sports image taken from a camera positioned around the playfield, the sports image including the sports participant within the playfield; Predicting the validity of an image-based location obtained by projecting the pixels of the sports participant in the sports image onto a reference plane having the same height as the playfield - wherein the predicting is performed to detect occlusion associated with the sports participant. comprising at least one of detecting vertical motion associated with the sports participant and calculating a change in the image-based position from the previous image-based position; if the image-based location is predicted to be valid: obtain the image-based location based on the pixel location of the sports participant in the sports image, and determine the location of the sports participant based on the image-based location; determining the location of the sports participant based on the positioning sensor-based location if the image-based location is predicted to be invalid.

A player tracking method disclosed in this application includes receiving a sensor signal from a positioning sensor located on a sports participant; obtaining a localization sensor-based position of the sports participant based on the sensor signal; obtaining a sports image captured by a camera located around a playfield, the sports image including the sports participant within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant within the sports image; calculate a disparity index between the positioning sensor-based position and the image-based position, the disparity index obtained from a difference between the positioning sensor-based position and the image-based position; determining a location of the sports participant based on a first location that is one of the positioning sensor-based location and the image-based location, if the separation index is less than a first predetermined threshold; if the separation index is greater than a second predetermined threshold: obtaining a first reliability index related to one of the positioning sensor-based position and the image-based position; obtaining a second reliability index associated with the other one of the positioning sensor-based location and the image-based location; if the first reliability index is greater than the second reliability index, determine the location of the sports participant based on the first location; and determining the location of the sports participant based on a second location, which is the other one of the positioning sensor-based location and the image-based location, when the first reliability index is less than the second reliability index.

A player tracking method disclosed in this application includes receiving a sensor signal from a positioning sensor located on a sports participant; obtaining a positioning sensor-based location of the sports participant based on the sensor signal; obtaining first reliability information related to the reliability of the location based on the positioning sensor; acquiring sports images taken from cameras positioned around the playfield, the sports images including the sports participants within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant within the sports image; Obtaining second reliability information related to the confidence of the image-based position - the confidence of the image-based position may include occlusion associated with the sports participant, vertical motion associated with the sports participant, and the image-based position from a previous image-based position. associated with at least one of the changes in base position; calculating weights based on the first reliability information and the second reliability information, wherein the weights include sensor-based weights and image-based weights; and

Calculating the location of the sports participant - the location of the sports participant may include obtaining from the positioning sensor-based position taking into account the sensor-based weights and the image-based position taking into account the image-based weights.

A player tracking method disclosed in this application includes receiving a sensor signal from a positioning sensor located on a sports participant; obtaining a positioning sensor-based location of the sports participant based on the sensor signal; preparing a reliability map corresponding to a playfield, wherein the reliability map includes a plurality of regions having reliability information including first reliability information related to the reliability of the location based on the positioning sensor; obtaining sports images taken from cameras positioned around the playfield, the sports images including the sports participants within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant in a singi sports image; determining a specific area occupied by the sports participant based on the positioning sensor-based location; determining a first weight and a second weight according to the first reliability information of the specific region; and determining the position of the sports participant based on a weighted average of the positioning sensor-based position and the image-based position calculated in consideration of the first weight and the second weight.

A player tracking method disclosed in the present application receives a plurality of sports images from a plurality of cameras installed at different locations around a playfield, each of the plurality of sports images including the playfield and at least a portion of the sports participants. ; detecting a target participant from the plurality of sports images; selecting at least one candidate image from the plurality of sports images according to a detection result of the target participant; detecting whether occlusion is associated with the target participant from the at least one candidate image; selecting at least one valid image from the at least one candidate image according to a result of the occlusion detection; and

and determining a position of the target participant based on a pixel position corresponding to the target participant of the at least one valid image.

A player tracking method disclosed in the present application receives a plurality of sports images from a plurality of cameras installed at different locations around a playfield, each of the plurality of sports images including the playfield and at least a portion of the sports participants. ; detecting a target participant from the plurality of sports images; selecting at least one candidate image from the plurality of sports images according to a detection result of the target participant; detecting whether occlusion is associated with the target participant from the at least one candidate image; selecting at least one target image from the at least one candidate image according to a result of the occlusion detection; When the occlusion is detected in all of the at least one candidate image, determining whether the sports participant occluded or was occluded by another sports participant in the at least one candidate image, and according to the determination select the target image from the at least one candidate image; and determining a location of the target participant based on a pixel location corresponding to the target participant of the at least one target image.

The player tracking method disclosed in this application obtains camera arrangement information including positions and poses of a plurality of cameras located around a playfield; receiving sensor signals from a plurality of positioning sensors located on a plurality of sports participants; obtaining locations of the plurality of sports participants based on the sensor signals; Creating a virtual playfield including a plurality of fixed points corresponding to the plurality of cameras and a plurality of moving points corresponding to the plurality of sports participants - the positions of the plurality of fixed points in the virtual playfield are approximately the same as the camera arrangement , where the locations of the plurality of moving points within the virtual playfield are determined based on the locations of the plurality of sports participants; calculating an association between the plurality of cameras and the plurality of sports participants based on relative positions between the plurality of fixed points and the plurality of moving points; predicting occlusion of the plurality of sports participants from the plurality of cameras based on angles between a plurality of virtual lines extending from the plurality of fixed points to the plurality of moving points and positions between the plurality of moving points; and generating a matching table between the plurality of cameras and the sports participant based on the relevance and the occlusion prediction.

According to an embodiment of the present specification, the player position can be more accurately tracked by calculating the player position by fusion of image data and positioning sensor data.

According to the embodiment of the present specification, the position of the player can be more accurately tracked by selecting image data optimized for the position of the player to be tracked using a plurality of cameras and calculating the position of the player therefrom.

1 is a diagram illustrating an example of a conventional positioning method.
2 is a diagram illustrating another example of a conventional positioning method.
3 is a schematic diagram of a player tracking system according to an embodiment of the present application.
4 is a block diagram of the player tracking server 1000 disclosed in this application.
5 to 8 are exemplary diagrams of a method of determining or detecting whether an occlusion event occurs between a plurality of players in image data of the player tracking server according to an embodiment of the present application.
9 is an exemplary diagram of a method for determining the validity of a player's position calculated from image data disclosed in the present application.
10 is an exemplary diagram of a method for determining the validity of a player's position calculated from image data disclosed in the present application.
11 is an exemplary diagram of a method for determining the validity of a player's position calculated from positioning sensor data disclosed in the present application.
12 is an exemplary diagram of a method for determining the validity of a player's position calculated from positioning sensor data disclosed in the present application.
13 is a diagram illustrating training of an artificial neural network for converting a position of a first coordinate system related to positioning sensor data into a position of a second coordinate system related to image data.
14 is a diagram illustrating conversion of a position of a first coordinate system related to positioning sensor data into a position of a second coordinate system related to image data using a learned artificial neural network.
15 is a diagram illustrating an exemplary method of determining a player location in a player tracking server according to an embodiment of the present application.
16 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application.
FIG. 17 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application, and is a diagram in which the step S1300 of FIG. 16 is subdivided.
18 is a flowchart illustrating a learning process of an artificial neural network related to a coordinate conversion algorithm according to an embodiment of the present application.
19 is a flowchart illustrating a method for tracking a player using coordinate transformation using a learned artificial neural network.
20 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application.
FIG. 21 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application, and is a diagram in which the step S4300 of FIG. 20 is subdivided.
22 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application.
23 is an exemplary diagram of a first reliability map according to an embodiment of the present application.
24 is an exemplary diagram of a second reliability map according to an embodiment of the present application.
25 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application.
26 is a schematic diagram illustrating a player tracking system using a plurality of image capture devices according to an embodiment of the present application.
27 is a diagram illustrating an exemplary process of selecting a valid image according to an embodiment of the present application.
28 is a flowchart illustrating a player tracking method using a plurality of image capture devices according to an embodiment of the present application.
29 is an exemplary diagram illustrating a virtual playfield according to an embodiment of the present application.
30 is a flowchart illustrating a method for tracking a player using a plurality of image capture devices according to an embodiment of the present application.
31 is a schematic diagram illustrating a method for calculating a location of a sports participant by a tracking method using a plurality of image capture devices according to an embodiment of the present application.

The foregoing objects, features and advantages of the present application will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, the present application can apply various changes and can have various embodiments. Hereinafter, specific embodiments will be illustrated in the drawings and described in detail.

Like reference numerals designate essentially like elements throughout the specification. In addition, components having the same function within the scope of the same idea appearing in the drawings of each embodiment will be described using the same reference numerals, and overlapping descriptions thereof will be omitted.

If it is determined that a detailed description of a known function or configuration related to the present application may unnecessarily obscure the subject matter of the present application, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of this specification are only identifiers for distinguishing one component from another component.

In addition, the suffixes "module" and "unit" for components used in the following embodiments are given or used interchangeably in consideration of ease of writing the specification, and do not have meanings or roles that are distinguished from each other by themselves.

In the following examples, expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean that features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added.

In the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present invention is not necessarily limited to those shown.

If an embodiment is otherwise implementable, the order of specific processes may be performed differently from the order described. For example, two processes that are described in succession may be performed substantially concurrently, or may proceed in an order reverse to that described.

In the following embodiments, when components are connected, a case in which the components are directly connected as well as a case in which components are interposed between the components and connected indirectly is included.

For example, when it is said that components are electrically connected in this specification, not only the case where the components are directly electrically connected, but also the case where the components are interposed and electrically connected indirectly is included.

According to the player tracking method disclosed in this application, a sensor signal is received from each of a plurality of positioning sensors located of a plurality of sports participants, the sensor signal including a participant identifier (idenfier) and location data; receiving captured sports images from cameras positioned around the playfield, the sports images including at least a target participant of a plurality of sports participants within the playfield; detecting, from the sports image, occlusion associated with the target participant; determining a severity of the occlusion based on the sensor signals received from specific positioning sensors mounted on specific sports players located in the region of interest related to the occlusion; if it is not detected that the severity of occlusion is greater than a threshold value, determining a location of the sports participant based on a pixel location associated with the target participant in the sports image; and if it is detected that the severity of the occlusion is greater than a threshold value, determining the location of the sports participant based on the location data of the sensor signal having the participant identifier representing the target participant.

According to the player tracking method disclosed in the present application, the region of interest is a region predetermined for the target participant involved in the occlusion, and determining the severity is based on at least one sensor signal located in the region of interest. and determining the severity of the occlusion based on the number.

According to the player tracking method disclosed in the present application, determining the severity is to assign a value representing serious occlusion to the severity when the number of signals of the at least one sensor is greater than the threshold value, and the at least one sensor and assigning a value representing mild occlusion to the severity if the number of signals is less than the threshold.

According to the player tracking method disclosed in the present application, the determining of the severity level is based on the participant identifier corresponding to the target participant and the other participant, so that the target participant and the other participant among at least one sports participant are in the same team. and obtaining team information indicating whether the other participant obscures the target participant or may be obscured by the target participant.

According to the player tracking method disclosed in the present application, determining the severity is to assign a value representing the serious occlusion to the severity, if the target participant and the other participant belong to the same team, and If the other participants belong to different teams, a value representing the mild occlusion may be assigned to the severity.

According to the player tracking method disclosed in the present application, the team information may be obtained from pixel data corresponding to the target participant and the other participant of the sports image.

According to the player tracking method disclosed in the present application, an image base representing a first position of a sports participant defined by a first coordinate system is obtained from sports images captured from cameras positioned around a playfield and including the sports participant within the playfield. obtaining location data, wherein the image-based location data is obtained based on a pixel location associated with the sports participant within the sports image; detecting an occlusion event related to the sports participant from the sports image; verifying the image-based location data based on a result of the detection; obtaining a positioning sensor-based position representing a second position of the sports participant defined by a second coordinate system based on a sensor signal obtained from a positioning sensor located on the sports participant; obtaining, based on the sensor signal or the second position, a localization sensor-based speed representing the speed of the sports participant defined by the second coordinate system; Preparing a neural network that transforms the position of the second coordinate system into the first coordinate system - The neural network includes an input layer for receiving position and speed according to the second coordinate system, an output layer for outputting a result representing a position value, and the including a hidden layer having a plurality of nodes connecting the input layer and the output layer; preparing a learning set, the learning set including the image-based positioning data determined to be valid and the positioning sensor-based position data and positioning sensor-based velocity data corresponding to the image-based positioning data determined to be valid; and inputting the positioning sensor-based position data and positioning sensor-based speed data of the learning set to the input layer using the learning set, and corresponding to the input positioning sensor-based position data and positioning sensor-based speed data. and learning the neural network by adjusting weights of the nodes based on a difference between the image-based location data and the location value of the result.

According to the player tracking method disclosed in the present application, the verifying determines that the image-based positioning data is valid if the occlusion event is not detected, and if the occlusion event is detected, the image-based positioning data may be determined to be invalid.

According to the player tracking method disclosed in the present application, the occlusion event indicates that the number of at least one sensor signal of at least one sports participant within a region of interest related to the occlusion is greater than a threshold, and the region of interest is occluded. The region where transition has occurred may be a predetermined region.

According to the player tracking method disclosed in this application, receiving sports images captured by cameras located around a playfield, the sports images including sports participants within the playfield; receiving a sensor signal from a positioning sensor mounted on the sports participant; detecting, from the sports image, an occlusion event related to the sports participant; if the occlusion event is not detected, determine a location of the sports participant based on image-based location data representing a location defined in a first coordinate system; When the occlusion event is detected: obtaining a positioning sensor-based position representing a position defined in a second coordinate system based on the sensor signal; obtaining a positioning sensor-based speed indicating a speed defined in the second coordinate system based on the sensor signal or the positioning sensor-based position; and determining a position of the sports participant defined in the first coordinate system from the positioning sensor-based position and the positioning sensor-based velocity using a neural network for converting a position in the second coordinate system to a position in the first coordinate system; The neural network includes an input layer for receiving position and speed according to the second coordinate system, an output layer for outputting a result indicating a position value according to the first coordinate system, and a plurality of layers connecting the input layer and the output layer. It includes a hidden layer having nodes, and can be trained by adjusting weights of the plurality of nodes using positions and velocities according to the second coordinate system labeled with positions according to the first coordinate system.

According to the athlete tracking method disclosed in this application, a sensor signal is received from a positioning sensor located on a sports participant; obtaining a positioning sensor based position based on the sensor signal; verifying the positioning sensor-based position based on a change of the positioning sensor-based position from a previous positioning sensor-based position; if the positioning sensor-based location is valid, determining the location of the sports participant based on the positioning sensor-based location; If the positioning sensor based location is not valid: obtaining a sports image taken from a camera positioned around the playfield, the sports image including the sports participant within the playfield; Predicting the validity of an image-based location obtained by projecting the pixels of the sports participant in the sports image onto a reference plane having the same height as the playfield - wherein the predicting is performed to detect occlusion associated with the sports participant. comprising at least one of detecting vertical motion associated with the sports participant and calculating a change in the image-based position from the previous image-based position; if the image-based location is predicted to be valid: obtain the image-based location based on the pixel location of the sports participant in the sports image, and determine the location of the sports participant based on the image-based location; determining the location of the sports participant based on the positioning sensor-based location if the image-based location is predicted to be invalid.

According to the player tracking method disclosed in the present application, verifying the positioning sensor-based position determines whether the positioning sensor-based position is determined when a change in the positioning sensor-based position from a previous positioning sensor-based position exceeds a predetermined threshold value. It is determined that the positioning sensor-based position is not valid, and when a change of the positioning sensor-based position from a previous positioning sensor-based position is less than a predetermined threshold value, the positioning sensor-based position may be determined to be valid.

According to the player tracking method disclosed in the present application, verifying the location based on the positioning sensor may include further considering information about reliability of the sensor signal included in the sensor signal.

According to the player tracking method disclosed in the present application, information about the reliability of the sensor signal included in the sensor signal may include at least one of DoP and SNR.

According to the player tracking method disclosed in the present application, a vertical movement related to the sports participant may be detected based on a movement of the sports participant in a direction perpendicular to the reference plane obtained from an inertial sensor worn by the sports participant. can

According to the athlete tracking method disclosed in this application, a sensor signal is received from a positioning sensor located on a sports participant; obtaining a positioning sensor-based location of the sports participant based on the sensor signal; acquiring sports images taken from cameras positioned around the playfield, the sports images including the sports participants within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant within the sports image; calculate a disparity index between the positioning sensor-based position and the image-based position, the disparity index obtained from a difference between the positioning sensor-based position and the image-based position; determining a location of the sports participant based on a first location that is one of the positioning sensor-based location and the image-based location, if the separation index is less than a first predetermined threshold; if the separation index is greater than a second predetermined threshold: obtaining a first reliability index related to one of the positioning sensor-based position and the image-based position; obtaining a second reliability index associated with the other one of the positioning sensor-based location and the image-based location; if the first reliability index is greater than the second reliability index, determine the location of the sports participant based on the first location; and determining the location of the sports participant based on a second location, which is the other one of the positioning sensor-based location and the image-based location, when the first reliability index is less than the second reliability index.

According to the athlete tracking method disclosed in this application, a sensor signal is received from a positioning sensor located on a sports participant; obtaining a positioning sensor-based position of the sports participant based on the sensor signal; obtaining first reliability information related to reliability of the location based on the positioning sensor; obtaining a sports image captured by a camera located around a playfield, the sports image including the sports participant within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant in a singi sports image; Obtaining second reliability information related to the confidence of the image-based position - the confidence of the image-based position may include occlusion associated with the sports participant, vertical motion associated with the sports participant, and the image-based position from a previous image-based position. associated with at least one of the changes in base position; calculating weights based on the first reliability information and the second reliability information, wherein the weights include sensor-based weights and image-based weights; and calculating the location of the sports participant - the location of the sports participant is obtained from the positioning sensor-based location taking into account the sensor-based weights and the image-based location taking into account the image-based weights.

According to the athlete tracking method disclosed in this application, a sensor signal is received from a positioning sensor located on a sports participant; obtaining a positioning sensor-based location of the sports participant based on the sensor signal; obtaining first reliability information related to the reliability of the location based on the positioning sensor; acquiring sports images taken from cameras positioned around the playfield, the sports images including the sports participants within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant within the sports image; Obtaining second reliability information related to the confidence of the image-based position - the confidence of the image-based position may include occlusion associated with the sports participant, vertical motion associated with the sports participant, and the image-based position from a previous image-based position. associated with at least one of the changes in base position; calculating weights based on the first reliability information and the second reliability information, wherein the weights include sensor-based weights and image-based weights; and

Calculating the location of the sports participant - the location of the sports participant may include obtaining from the positioning sensor-based position taking into account the sensor-based weights and the image-based position taking into account the image-based weights.

According to the player tracking method disclosed in the present application, the reliability of the positioning sensor-based position may be calculated by calculating a change in the positioning sensor-based position from a positioning sensor-based position at a previous time and reliability of the sensor signal included in the sensor signal. It may be related to at least one of the information about.

According to the player tracking method disclosed in the present application, the vertical motion associated with the sports participant is the sports participant in a vertical direction with respect to a reference plane having the same height as the play field obtained from an inertial sensor worn by the sports participant. It can be detected based on the movement of the participant.

According to the player tracking method disclosed in the present application, the reliability of the positioning sensor-based location and the reliability of the image-based location are related to a disparity index between the positioning sensor-based location and the image-based location, and the disparity index is It may be obtained from a difference between the positioning sensor-based position and the image-based position.

According to the athlete tracking method disclosed in this application, a sensor signal is received from a positioning sensor located on a sports participant; obtaining a positioning sensor-based location of the sports participant based on the sensor signal; preparing a reliability map corresponding to a playfield, wherein the reliability map includes a plurality of regions having reliability information including first reliability information related to the reliability of the location based on the positioning sensor; obtaining sports images taken from cameras positioned around the playfield, the sports images including the sports participants within the playfield; obtaining an image-based location of the sports participant based on the pixel location of the sports participant within the sports image; determining a specific area occupied by the sports participant based on the positioning sensor-based location; determining a first weight and a second weight according to the first reliability information of the specific region; and determining the position of the sports participant based on a weighted average of the positioning sensor-based position and the image-based position calculated in consideration of the first weight and the second weight.

According to the player tracking method disclosed in the present application, the first reliability information related to the reliability of the positioning sensor-based position may include calculating a change in the positioning sensor-based position from a positioning sensor-based position at a previous point in time and the sensor signal. It may be related to at least one of included information about the reliability of the sensor signal.

According to the player tracking method disclosed in the present application, the reliability map may include a plurality of regions having reliability information including second reliability information related to the reliability of the image-based location.

According to the player tracking method disclosed in the present application, the second reliability information related to the reliability of the image-based position may include detecting occlusion related to the sports participant and detecting vertical motion related to the sports participant. and calculating a change in the image-based location from a previous image-based location.

According to the player tracking method disclosed in the present application, the first weight and the second weight may be determined according to the second reliability information of the specific region.

According to the player tracking method disclosed in the present application, a plurality of sports images are received from a plurality of cameras installed at different locations around a playfield, each of the plurality of sports images including the playfield and at least a portion of the sports participants. -; detecting a target participant from the plurality of sports images; selecting at least one candidate image from the plurality of sports images according to a detection result of the target participant; detecting whether occlusion is associated with the target participant from the at least one candidate image; selecting at least one valid image from the at least one candidate image according to a result of the occlusion detection; and

and determining a position of the target participant based on a pixel position corresponding to the target participant of the at least one valid image.

According to the player tracking method disclosed in the present application, if the at least one valid image includes a first valid image and a second valid image, determining the location of the target participant is based on the target participant in the first valid image. obtaining a first location of the target participant based on the corresponding pixel location; obtaining a second position of the target participant based on a pixel position corresponding to the target participant in the second valid image; and calculating a location of the target participant based on both the first location and the second location.

According to the player tracking method disclosed in the present application, determining the location of the target participant includes a first condition related to a location between the target participant and a camera that captures the at least one valid image and captures the at least one valid image. selecting a single image from the at least one valid image based on a second condition related to a lens distortion parameter of a camera and calculating the position of the target participant based on the single image.

According to the player tracking method disclosed in the present application, a plurality of sports images are received from a plurality of cameras installed at different locations around a playfield, each of the plurality of sports images including the playfield and at least a portion of the sports participants. -; detecting a target participant from the plurality of sports images; selecting at least one candidate image from the plurality of sports images according to a detection result of the target participant; detecting whether occlusion is associated with the target participant from the at least one candidate image; selecting at least one target image from the at least one candidate image according to a result of the occlusion detection; When the occlusion is detected in all of the at least one candidate image, determining whether the sports participant occluded or was occluded by another sports participant in the at least one candidate image, and according to the determination select the target image from the at least one candidate image; and determining a location of the target participant based on a pixel location corresponding to the target participant of the at least one target image.

According to the athlete tracking method disclosed in the present application, the at least one target image may be selected from the at least one candidate image in which the sports participant obscures another sports participant in the at least one candidate image.

According to the player tracking method disclosed in the present application, if the at least one target image includes a first target image and a second target image, determining the location of the target participant is based on the target participant in the first target image. obtaining a first location of the target participant based on the corresponding pixel location; obtaining a second location of the target participant based on a pixel position corresponding to the target participant in the second target image; and calculating a location of the target participant based on both the first location and the second location.

According to the player tracking method disclosed in the present application, determining the location of the target participant includes a first condition related to a location between the target participant and a camera that captures the at least one target image and captures the at least one target image. selecting a single image from the at least one target image based on a second condition related to a lens distortion parameter of a camera and calculating a position of the target participant based on the single image.

According to the player tracking method disclosed in this application, obtaining camera arrangement information including positions and poses of a plurality of cameras located around the playfield; receiving sensor signals from a plurality of positioning sensors located on a plurality of sports participants; obtaining locations of the plurality of sports participants based on the sensor signals; Creating a virtual playfield including a plurality of fixed points corresponding to the plurality of cameras and a plurality of moving points corresponding to the plurality of sports participants - the positions of the plurality of fixed points in the virtual playfield are approximately the same as the camera arrangement , where the locations of the plurality of moving points within the virtual playfield are determined based on the locations of the plurality of sports participants; calculating an association between the plurality of cameras and the plurality of sports participants based on relative positions between the plurality of fixed points and the plurality of moving points; predicting occlusion of the plurality of sports participants from the plurality of cameras based on angles between a plurality of virtual lines extending from the plurality of fixed points to the plurality of moving points and positions between the plurality of moving points; and generating a matching table between the plurality of cameras and the sports participant based on the relevance and the occlusion prediction.

See Figure 1. 1 is a diagram illustrating an example of a conventional positioning method. Specifically, FIG. 1 is a diagram showing a specific situation of a positioning method for calculating a point where a player is located from an image captured through a camera.

According to the positioning method using an image, the player's position can be accurately calculated only when the player to be tracked within the image is recognized.

For example, with respect to a player located in the first area 10 of FIG. 1 , the player may be recognized and the position of the player to be tracked may be calculated.

However, for a player located in the second region 12 of FIG. 1 , an occlusion event that makes it difficult to recognize the player may occur. Specifically, since a plurality of players are concentrated in the second region 12, occlusion may occur in which a player to be tracked is covered by other players in the image. Therefore, the location of the player to be tracked hidden by other players may not be accurately obtained.

In other words, the positioning method using an image may not be able to cope with an obstruction situation, such as a player to be tracked in the image being covered by another player.

See Figure 2. 2 is a diagram illustrating an example of a conventional positioning method. Specifically, FIG. 2 is a diagram illustrating a positioning method for calculating a point where a player is located using positioning instruments such as a GPS module.

Specifically, the positioning method using the GPS module calculates the position of the player to be tracked depending on the signal transmitted from the satellite. However, signals transmitted from satellites may be greatly affected by structures around the athlete to be tracked.

For example, GPS signals transmitted from some of the satellites 24 and 26 of FIG. 2 can be transmitted inside the stadium without being affected by surrounding structures of a player to be tracked. However, GPS signals transmitted from some of the satellites 22 and 28 in FIG. 2 may not reach the inside of the stadium due to the influence of surrounding structures of the player to be tracked. At this time, if the GPS signals transmitted from some of the satellites 22 and 28 are affected by surrounding structures, an error may occur in the position of the player calculated from the GPS signals.

In addition, since the degree to which the GPS signal is affected by surrounding structures is different for each area of the playfield in the stadium, the positioning method using the GPS module has a problem in that the reliability is different for each area of the playfield in the stadium. .

In this application, a player tracking device, a player tracking system, and a player tracking method for overcoming the limitations of the above-described positioning method using an image and positioning method using a positioning mechanism such as a GPS or LPS module will be described.

Hereinafter, the player tracking device, player tracking system, and player tracking method of the present application will be described. Here, player tracking may be performed using positioning sensor data obtained from a positioning sensor mounted on the player or image data obtained through at least one camera. The present application discloses techniques such as fusion of the above-described data or use of multiple image data in order to improve the accuracy of the player's position calculated from such basic data.

Hereinafter, a player tracking method, player tracking device, and player tracking system according to an embodiment of the present application will be described.

Player tracking according to the present embodiment may be performed using both image data and positioning sensor data. Player tracking using image data can calculate the player's position based on pixel data related to the player's position in the image data, and player tracking using positioning sensor data provides latitude and longitude related to the player's position included in the positioning sensor data. The position of the player may be calculated based on data such as the back. At this time, if the tracking of the player using the image data is inaccurate, the tracking of the player can be supplemented by the positioning sensor data, and if the tracking of the player using the positioning sensor data is inaccurate, the tracking of the player can be supplemented by the image data. can do.

3 is a schematic diagram of a player tracking device, a player tracking system, and a player tracking method according to an embodiment of the present application.

Referring to FIG. 3 , the player tracking device, player tracking system, and player tracking method according to the present embodiment may track a player position by using both image data and positioning sensor data.

Hereinafter, the player tracking system 100 according to an embodiment of the present application will be described with reference to FIG. 3 .

Referring to FIG. 3 , the player tracking system 100 may include a positioning sensor device 200 , an image capture device 300 , an intermediate server 400 and a player tracking device 1000 .

The positioning sensor device 200 of the player tracking system 100 may be a positioning device such as a GPS module or an LPS module.

For example, the positioning sensor device 200 may be a GPS module 210, and the GPS module receives radio waves transmitted from satellites in relation to a global navigation system (GNSS) to locate the positioning sensor device 200. Data for the player may be transmitted to the tracking device (1000).

As another example, the positioning sensor device 200 may be the LPS module 220, and the LPS module receives radio waves transmitted from a beacon (beacon, 4) or the like in relation to a local positioning system (LPS), Data on the position of the positioning sensor device 200 may be transmitted. The beacon 4 may be located at the periphery of the playfield, and its location may be different.

The image capture device 300 of the player tracking system 100 may be a camera of any suitable form and function. The image capture device 300 may transmit image data acquired by photographing the sports participant 1 , the playfield line 2 , and the playfield 3 to the player tracking device 1000 .

The player tracking device 1000 of the player tracking system 100 may acquire image data and positioning sensor data. Also, the player tracking device 1000 may obtain an image-based position from image data and a positioning sensor-based position from positioning sensor data. In addition, the player tracking apparatus 1000 may determine the validity of the image-based position and the positioning sensor-based position to determine the position of the sports participant with high accuracy.

The player tracking device 1000 may be implemented as any suitable type of server device. In the following description, the player tracking device 1000 is referred to as the player tracking server 1000, but this is merely for convenience of explanation, and the player tracking device 1000 does not necessarily have to be implemented in the form of a server.

Meanwhile, the player tracking system 100 may further include an intermediate server 400 or a repeater. In other words, the player tracking server 1000 of the player tracking system 100 may be provided in a single form, but may be provided in a separate form rather than a single form.

At this time, the data transmitted from the positioning sensor device 200 and the image capture device 300 may be directly transmitted to the player tracking server 1000 or indirectly transmitted to the player tracking server 1000 via the intermediate server 400. It could be. When passing through the intermediate server 400 , data may be transferred to the portable electronic device 420 through the docking station 410 . Data may also be transmitted from the portable electronic device 420 to the player tracking server 1000 using any suitable communication method.

Hereinafter, the player tracking server 1000 according to an embodiment of the present application will be described.

According to one embodiment of the present application, the player tracking server 1000 may obtain image data from the image capture device 300 to determine the location of a sports participant. Also, the player tracking server 1000 may obtain positioning sensor data from a positioning sensor device to determine the position of a sports participant. Also, the player tracking server 1000 may calculate an image-based position from the obtained image data. Also, the player tracking server 1000 may calculate a positioning sensor-based position from the obtained positioning sensor data. Additionally, the player tracking server 1000 may determine the position of the sports participant based on the calculated image-based position and positioning sensor-based position.

See Figure 4. 4 is a block diagram of a player tracking server 1000 according to an embodiment of the present application.

Referring to FIG. 4 , the player tracking server 1000 may include a communication module 1100 , a memory 1200 and a controller 1300 .

Hereinafter, each component of the player tracking server 1000 according to an embodiment of the present application will be described.

The communication module 1100 may communicate with an external device. The player tracking server 1000 may transmit/receive data with an external server including the image capture device 300, the positioning sensor device 200, or the intermediate server 400 through the communication module 1100.

For example, the player tracking server 1000 may receive image data taken from the image capture device 300 or positioning sensor data obtained from the positioning sensor device 200 through the communication module 1100 . As another example, the player tracking server 1000 may access the Internet through the communication module 1100 and upload image data or positioning sensor data.

The communication module 1100 is largely divided into a wired type and a wireless type. Since the wired type and the wireless type each have advantages and disadvantages, the wired type and the wireless type may be simultaneously provided in the player tracking server 1000 in some cases.

Here, in the case of the wired type, LAN (Local Area Network) or USB (Universal Serial Bus) communication is a representative example, and other methods are also possible.

Also, in the case of the wireless type, a wireless personal area network (WPAN) communication method such as Bluetooth or Zigbee may be used. However, since the wireless communication protocol is not limited thereto, the wireless type communication module may use a wireless local area network (WLAN)-based communication method such as Wi-Fi or other known communication methods.

The memory 1200 may store various kinds of information. Various types of data may be temporarily or semi-permanently stored in the memory 1200 . Examples of the memory 1200 include a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a read-only memory (ROM), a random access memory (RAM), and the like. This can be.

The memory 1200 may be provided in a form embedded in the player tracking server 1000 or in a detachable form. The memory 1200 includes an operating system (OS) for driving the player tracking server 1000 or a program for operating each component of the player tracking server 1000, as well as necessary for the operation of the player tracking server 1000. Various types of data may be stored. For example, the memory 1200 may store image data in the form of an intensity map and an RGB map obtained from the image capture device 300 and positioning sensor data obtained from the positioning sensor device 200 .

The controller 1300 may control overall operations of the player tracking server 1000 . For example, the controller 1300 may load and execute a program for operating the player tracking server 1000 from the memory 1200 .

The controller 1300 may be implemented as a central processing unit (CPU) or a device similar thereto according to hardware or software or a combination thereof. It may be provided in the form of an electronic circuit that performs a control function by processing electrical signals in hardware, and may be provided in the form of a program or code that drives a hardware circuit in terms of software.

Hereinafter, several operations performed by an embodiment of the player tracking system 100 in relation to the player tracking method will be described in more detail.

The player tracking system 100 according to an embodiment of the present application may obtain image data and positioning sensor data.

The player tracking server 1000 may acquire image data and positioning sensor data. Specifically, the player tracking server 1000 may acquire image data from the image capture device 300 through the communication module 1100 . Also, the player tracking server 1000 may obtain information related to the image capture device 300 from the image capture device 300 through the communication module 1100 . Also, the player tracking server 1000 may obtain positioning sensor data from the positioning sensor device 200 through the communication module 1100 .

The image data may include pixel information related to a plurality of sports participants, lines and dots of a playfield, balls, etc., and sports game progress information (elapsed time, image data capture time).

Information related to the image capture device 300 may include arrangement information related to a position and pose of a camera. Alternatively, among the information related to the image capture device 300, the player tracking server 1000 may obtain arrangement information related to the position of the camera by any suitable method.

The positioning sensor data may include data related to the location of the positioning sensor device and data related to reliability of the sensor-signal.

The player tracking system 100 according to an embodiment of the present application may resample image data or remove noise.

Specifically, the player tracking server 1000 may be controlled to perform an operation of resampling the image data. Specifically, the player tracking server 1000 may perform an operation of resampling image data to a specific frame. Through the resampling operation of the player tracking server 1000, time synchronization with positioning sensor data to be described later may be matched. A particular frame may be any suitable frame (eg, 10 frames).

Also, the player tracking server 1000 may be implemented to remove noise from the image data. Specifically, image data photographed by the image capture device may have distortion. For example, there may be distortion of image data due to distortion of a lens, distortion of image data according to a pose (eg, angle) of a camera, or a distance from the camera. Accordingly, the player tracking server 1000 may perform an operation of correcting distortion of the image data described above.

The player tracking system 100 according to an embodiment of the present application may perform an operation of correlating image data and positioning sensor data. Specifically, the player tracking system 100 may time-synchronize image data and positioning sensor data.

Specifically, the controller 1300 of the player tracking server 1000 may be provided to correlate the image data and the positioning sensor data using a time variable. In other words, the controller 1300 of the player tracking server 1000 may be implemented to match time synchronization between the image data and the positioning sensor data.

For example, time synchronization between the image data and the positioning sensor data may be based on a specific point in time of a sports game.

In this case, the specific viewpoint of the sports game may be a characteristic viewpoint of the sports game. For example, the specific point in time of the sports game may be the start point of the game.

Also, when there is a break time in the sports game, the specific time point of the sports game may be a game restart time point. For example, in the case of a soccer game, there is half time between the first half and the second half. In this case, the time synchronization between the image data and the positioning sensor data may be set based on the start time of the first half and the start time of the second half. At the start time of the first half and the start time of the second half, a certain number of sports participants start moving positions, and accordingly, a change in speed of the sports participant may occur. Accordingly, time synchronization between the image data and the positioning sensor data may be achieved in consideration of a change in position or speed of a sports participant between the start of the first half and the start of the second half.

For example, based on the point in time when the speed data of at least one sports participant obtained or calculated from the image data and the speed data of at least one sports participant obtained or calculated from the positioning sensor data match. Time synchronization between the image data and the positioning sensor data may be matched.

In this case, the number of sports participants to be compared between the speed data from the image data and the speed data from the positioning sensor data may be any appropriate number. In other words, time synchronization through the above-described comparison of speed data may be performed not only for all sports participants, but also for some sports participants.

For example, time synchronization between the image data and the positioning sensor data may be achieved in consideration of a time point at which a speed change obtained from the positioning sensor data is detected. For example, after a predetermined time has elapsed after the start of the first half and the second half, most of the sports participants start to move, and accordingly, the speed of the sports participants may change. The positioning sensor device may obtain data related to position changes of the sports participants, and a start point of a speed change obtained from the data related to position changes may be detected. According to an example, time synchronization between the image data and the positioning sensor data may be matched based on a specific point in time (eg, greater than 0 seconds to less than 2 seconds) before the point in time at which the change in speed is detected.

In this case, in relation to the start time of the speed change, it may be determined that the speed change starts when a speed change of sports participants of a predetermined threshold or more is detected.

In addition, an appropriate calculation method such as a Wasserstein distance calculation method may be used to match the time synchronization.

Also, for accuracy, time synchronization between the image data and the positioning sensor data needs to be synchronized not only at the start of the game but also during the game.

According to an embodiment of the present application, based on a point in time at which a positional relationship between a plurality of sports participants obtained from image data at an arbitrary time point during a game and a positional relationship between a plurality of sports participants obtained from positioning sensor data correspond Time synchronization between image data and positioning sensor data may be matched.

For example, the positioning sensor data may include an identification factor of each sports participant, and position data between at least one or more sports participants identified by the identification factor may be respectively obtained from the positioning sensor data.

Also, in the image data, each of the at least one sports participant identified in the positioning sensor data may be distinguished and identified, and position data between the identified at least one or more sports participant may be respectively obtained from the image data.

At this time, a connection relationship between sports participants according to the location data of at least one sports participant obtained from the positioning sensor data may be obtained, and also according to the location data of at least one sports participant obtained from the image data. Connections between sports participants can be obtained. In this case, time synchronization between the image data and the positioning sensor data may be performed based on a point in time when a connection relationship between sports participants obtained from the positioning sensor data and a connection relationship between sports participants obtained from the image data match or are within an error range. For example, the connection relationship between the sports participants may be expressed as a line or a plane connecting location data between sports participants obtained from image data or positioning sensor data.

When the connection relationship between the sports participants is a line or a line, time synchronization between the image data and the positioning sensor data may be achieved in consideration of whether the lengths or directions of the lines match.

When the connection relationship between the sports participants is flat, time synchronization between image data and positioning sensor data may be matched in consideration of whether shapes of figures composed of the outlines of the faces match.

The player tracking system 100 according to an embodiment of the present application may perform any appropriate data processing such as noise removal, amplification, and filtering on the positioning sensor data acquired from the positioning sensor device 200 . Specifically, it may be provided to perform the above-described data processing by the player tracking server 1000.

The player tracking system 100 according to an embodiment of the present application may convert the format of image data. Specifically, the player tracking server 1000 may convert the shape of image data received from the image capture device 300 from an RGB map to an intensity map. In this case, data related to the RGB map may be stored in the memory 1200 of the player tracking server 1000. Data related to the RGB map of the image data stored in the memory 1200 of the player tracking server 1000 is used when the player tracking server 1000, which will be described later, detects an occlusion event between players or determines the severity of an occlusion event. can be used for By converting these image data types, the image data is basically tracked in the form of an intensity map, but since the data included in the RGB map can be used only when necessary, the amount of computation of the player tracking server 1000 is reduced. can be reduced In addition, since the amount of calculation is reduced, calculation of player tracking can be faster.

The player tracking system 100 according to an embodiment of the present application may recognize and detect the playfield 3 included in image data. Specifically, the player tracking server 1000 may perform an operation of recognizing and detecting a playfield included in image data in consideration of a reference point of the playfield.

For example, the player tracking server 1000 may use reference points such as a corner flag, a half line, a goal line, a touch line, and a penalty mark of a playfield. It is possible to recognize and detect the play field included in the image data by taking into account. However, the reference point described above is only an example, and a playfield included in image data can be recognized and detected by considering any appropriate area, line, point, etc. in a playfield capable of recognizing and detecting a playfield.

The player tracking system 100 according to an embodiment of the present application may perform an image stitching operation.

Specifically, the player tracking server 1000 of the player tracking system 100 may provide image matching from one or more image data to single image data.

For example, image data may be obtained from at least one or more image capture devices disposed at different locations around the playfield. At this time, the player tracking server 1000 may generate a high-resolution image by performing an operation of matching at least one image data obtained from at least one image capture device of the same temporal viewpoint into one image data, and may generate a panoramic image. can create In this case, when matching at least one image data into one image data, matching operation between the image data may be performed based on a reference point included in an overlapping portion between the at least one or more image data. The "reference point" included in the overlapping portion between one or more image data may be a point such as a penalty mark or a corner flag included in the image data. Alternatively, the "reference point" included in the overlapping portion between the plurality of image data may be a line such as a penalty line, a goal line, a touch line, or a center line included in the image data. The "reference point" included in the overlapping portion between the plurality of image data may be a region such as a center circle included in the image data. However, this is only an example and any suitable reference point may be used to perform a matching operation between a plurality of image data.

The player tracking system 100 according to an embodiment of the present application may obtain an image-based location from image data.

Specifically, the player tracking server 1000 may analyze the image data and calculate the location of the sports participant included in the image data.

For example, the player tracking server 1000 may convert image data in a perspective view form into a top view form in order to calculate the position of a sports participant. In this case, the player tracking server 1000 may consider an installation location of an image capture device that captures image data or a relative location between sports participants from the image capture device in order to convert coordinates into a top view form. In addition, the player tracking server 1000 may convert image data in a perspective view form to image data in a top view form based on substantially the same plane as the playfield.

The player tracking server 1000 may calculate the position of a sports participant from top view image data. For example, the player tracking server 1000 may calculate the position of the sports participant based on the position of a pixel corresponding to the sports participant in the top view image data. Preferably, the player tracking server 1000 may calculate the position of the sports participant based on the position of a pixel corresponding to the foot of the sports participant.

As described above, the player tracking server 1000 mainly describes the content of calculating the player's position based on the top-view image data, but this is only an example. For example, the player tracking server 1000 acquires the position of a pixel corresponding to a sports participant (eg, foot) of image data (eg, a perspective view) in a form other than a top view, and then acquires an image capturing device. The player's position can be calculated by projecting it on the same plane as the playfield, considering the height and installation angle of the player.

As another example, various artificial neural networks such as CSRT, fast RCNN, and Depp sort may be used to track the sports participant by calculating the position of the sports participant by analyzing the image data. However, the artificial neural network described above is only an example, and any appropriate algorithm or artificial neural network may be used to track a sports participant.

For example, sports participants included in image data may be segmented through the above-described artificial neural network, and sports participants may be detected. If a sports participant is detected, a bounding box may be created, and the player tracking server 1000 may calculate the location of the sports participant based on the bounding box. Preferably, the player tracking server 1000 may calculate the position of the sports participant based on the bottom line of the bounding box.

As another example, the "foot" of the sports participant included in the image data may be detected through the above-described artificial neural network, and the player tracking server 1000 determines that the sports participant is based on the location of a pixel corresponding to the "foot". A sports participant's location may be calculated.

The player tracking system 100 according to an embodiment of the present application may obtain a positioning sensor-based location from positioning sensor data. Specifically, the player tracking server 1000 may calculate the player's position by considering data related to the player's position included in the positioning sensor data. For example, the player tracking server 1000 may calculate or obtain a positioning sensor-based position through triangulation based on positioning sensor data.

Here, the positioning sensor data includes data related to the player's position. In this case, or the positioning sensor-based position may be obtained from data related to speed or acceleration included in the positioning sensor data.

In addition, the player tracking server 1000 is provided to calculate the speed or acceleration of the sports participant using a mathematical method such as calculus from data related to the position of the above-described image data and data related to the position of the positioning sensor data. can

The player tracking system 100 according to an embodiment of the present application may determine the validity of the obtained image-based location.

Specifically, the player tracking server 1000 may evaluate the validity of the player's image-based position calculated from image data. More specifically, the player tracking server 1000 evaluates the validity of the image-based position, whether or not an occlusion event is detected, the severity of the occlusion event, the player's vertical movement, and calculated from the image data. It may be provided to take into account at least one of the internal disparity of the player's position.

5 to 8. FIGS. 5 to 8 are a method for determining or detecting whether an occlusion event occurs between a plurality of players in image data of the player tracking server 1000 according to an embodiment of the present application. It is an exemplary drawing for

For example, the player tracking server 1000 may evaluate the validity of the player's position calculated from the image data in consideration of whether an occlusion event is detected between a plurality of players in the image data. Here, the occlusion event may mean an event in which at least a part overlaps between a plurality of players.

For example, the player tracking server 1000 may detect an occlusion event based on image data in the form of an intensity map or an RGB map. For example, the player tracking server 1000 may recognize sports participants included in the image data based on any suitable artificial intelligence network as described above, and as a result of the recognition, a bounding box for each sports participant is formed. can be created At this time, the player tracking server 1000 may determine whether or not occlusion has occurred in consideration of overlapping levels of bounding boxes between a plurality of players.

For example, a threshold related to the degree of overlap of bounding boxes may be set in advance, and the player tracking server 1000 may exceed the threshold and overlap the bounding boxes between players. It can be determined that an occlusion event has occurred. On the other hand, the player tracking server 1000 may determine that no occlusion event has occurred when bounding boxes between players overlap by less than the threshold or when bounding boxes do not overlap. .

Specifically, refer to FIG. 5 . FIG. 5 shows the size or ratio of overlapping regions of a first bounding box BB1 corresponding to a first sports participant recognized from image data and a second bounding box BB2 corresponding to a second sports participant recognized from image data. It is a diagram illustrating detecting or determining whether or not occlusion is performed according to.

Referring to FIG. 5(a) , there may be a situation in which the first bounding box BB1 and the second bounding box BB2 do not overlap. At this time, the player tracking server 1000 may determine that no occlusion event has occurred.

Referring to FIGS. 5(b), (c), and (d), a situation in which the first bounding box BB1 and the second bounding box BB2 partially overlap may exist. At this time, the player tracking server 1000 performs occlusion based on a preset first threshold value for the size or ratio of the overlapping area of the first bounding box BB1 and the second bounding box BB2. It is possible to determine whether an event has occurred.

For example, FIG. 5( b ) may be a diagram illustrating a case where the size or ratio of the overlapping area of the first bounding box BB1 and the bounding box BB2 is smaller than a preset first threshold value. . At this time, the player tracking server 1000 may determine that no occlusion event has occurred between the first sports participant and the second sports participant.

As another example, FIG. 5(c) is a diagram illustrating a case where the size or ratio of the overlapping area of the first bounding box BB1 and the bounding box BB2 exceeds a first preset threshold value. can At this time, the player tracking server 1000 may determine that an occlusion event has occurred between the first sports participant and the second sports participant.

According to the foregoing, it has been described that the player tracking server 1000 can determine "whether" an occlusion event has occurred, but the player tracking server 1000 additionally considers a preset second threshold value to determine the first A “severity” of the occlusion event may be determined between the sports participant and the second sports participant.

For example, FIG. 5(c) may be a diagram illustrating a case where the size or ratio of the overlapped region of the first bounding box BB1 and the bounding box BB2 is smaller than the preset second threshold value. there is. In other words, in the case of FIG. 5(c), the player tracking server 1000 may determine that an occlusion event has occurred between the first sports participant and the second sports participant, but the severity of the occlusion is not serious.

On the other hand, FIG. 5(d) illustrates a case in which the size or ratio of the overlapped region of the first bounding box BB1 and the bounding box BB2 is greater than the first threshold value and the second threshold value set in advance. It may be a drawing shown.

At this time, the player tracking server 1000 may determine that an occlusion event has occurred between the first sports participant and the second sports participant, and that the severity of the occlusion is serious.

As another example, the player tracking server 1000 may detect an occlusion event based on image data in the form of an RGB map (or an intensity map). Specifically, the player tracking server 1000 may determine whether an occlusion event has occurred by detecting an occlusion event using pixel data included in the RGB map.

For example, pixels corresponding to the plurality of players included in the image data at the current time relative to the sum of the number of pixels corresponding to the plurality of players included in the image data at the previous time when the occlusion event did not occur. When the sum of the numbers decreases below a predetermined rate, the player tracking server 1000 may be implemented to determine that an occlusion event has occurred. For this purpose, an arbitrary appropriate threshold ratio may be set in advance with respect to the predetermined ratio.

Specifically, referring to FIG. 6 , the player tracking server 1000 may obtain pixel data related to each of the first and second sports participants from image data. At this time, the player tracking server 1000 detects whether an occlusion event occurs between the first and second sports participants in consideration of the pixel data related to the first sports participant and the pixel data related to the second sports participant, or can judge

Specifically, FIG. 6(a) is a diagram illustrating a plurality of pixel data PD1 corresponding to a first sports participant and a plurality of pixel data PD2 corresponding to a second sports participant. In this case, the plurality of pixel data PD1 corresponding to the first sports participant and the plurality of pixel data PD2 corresponding to the second sports participant may not overlap.

6(b) is a diagram illustrating a plurality of pixel data PD3 in which a part of a plurality of pixel data PD1 corresponding to a first sports participant and a plurality of pixel data PD2 corresponding to a second sports participant are overlapped. am.

According to a preferred embodiment, FIGS. 6(a) and 6(b) may be a plurality of pixel data related to a first sports participant and a second sports participant obtained from image data for two temporally adjacent views. At this time, the player tracking server 1000 calculates the sum of the plurality of pixel data PD1 corresponding to the first sports participant and the plurality of pixel data PD2 corresponding to the second sports participant in FIG. 6 (a). , it is possible to detect or determine whether an occlusion event has occurred in consideration of the ratio to the total sum of the plurality of pixel data PD3 of FIG. 6(b).

For example, the plurality of pixel data PD1 corresponding to the first sports participant and the plurality of pixel data PD2 corresponding to the second sports participant in FIG. 6(b) ( A reference threshold ratio value for whether an occlusion event occurs with respect to the ratio to the total sum of PD3) may be set in advance. At this time, the player tracking server 1000 calculates the sum of the plurality of pixel data PD1 corresponding to the first sports participant and the plurality of pixel data PD2 corresponding to the second sports participant, as shown in FIG. 6(b). When the ratio to the total of the plurality of pixel data PD3 is smaller than the reference threshold ratio, it may be determined that an occlusion event between the first sports participant and the second sports participant has occurred.

However, the above description is only an example, and it is possible to determine whether an occlusion event has occurred using any appropriate method using pixel data. In addition, it is obvious that the player tracking server 1000 can determine not only whether an occlusion event has occurred but also the severity of the occlusion event by setting an additional threshold ratio in advance.

As another example, the player tracking server 1000 may detect the occlusion event based on considering the data on the position of the player included in the positioning sensor data. Specifically, the player tracking server 1000 may obtain data related to positions of a plurality of players included in positioning sensor data corresponding to the plurality of players. At this time, the player tracking server 1000 may determine whether an occlusion event occurs by considering the number of players located within a predetermined radius based on a specific player. Regarding the number of players located within the predetermined radius, any suitable threshold may be set in advance, and the player tracking server 1000 determines the number of players located within a predetermined radius based on a specific player. If it is greater than the value, it may be determined that an occlusion event has occurred. On the other hand, when the number of players located within a predetermined radius based on a specific player is smaller than the threshold value, the player tracking server 1000 may determine that an occlusion event has occurred.

See FIG. 7 . 7 is a diagram illustrating locations of a plurality of sports participants.

The positions of the plurality of sports participants may be positions obtained from image data or positioning sensor data. Preferably, the positions of each of the plurality of sports participants may be positions obtained from positioning sensor data. Also, preferably, the positions of the plurality of sports participants may be positions obtained by coordinate conversion of data related to positions included in positioning sensor data. At this time, the player tracking server 1000 determines the number of sports participants located in an area R within a predetermined radius based on a specific location (L1_sensor) corresponding to a specific sports participant among locations corresponding to the plurality of sports participants. number can be obtained. In addition, the player tracking server 1000 may determine whether an occlusion event has occurred in consideration of the number of sports participants located in the area R within a predetermined radius based on the specific location L1_sensor. . For example, a threshold value may be set in advance in relation to the number of sports participants located in the area R within a predetermined radius, and the player tracking server 1000 determines the predetermined location based on the specific location L1_sensor. When the number of sports participants located in the area R within the radius is greater than the threshold value, it may be determined that an occlusion event has occurred. On the other hand, the player tracking server 1000 determines that no occlusion event has occurred when the number of sports participants located in the area R within a predetermined radius based on the specific location L1_sensor is smaller than the threshold value. can do.

As another example, the player tracking server 1000 may acquire data related to the arrangement and pose of the image capture device. In addition, the player tracking server 1000 may also receive positioning sensor data including data related to the position of each player from the positioning sensor device 200 worn by each player. In this case, the player tracking server 1000 may detect whether or not occlusion occurs by considering both positioning sensor data including data related to the arrangement and pose of the image capture device and data related to the position of the player. For example, the player tracking server 1000 detects whether or not occlusion has occurred based on the position of the player calculated from data related to the line of sight and angle of the image capture device and positioning sensor data. or can be judged.

See FIG. 8 . 8 relates to a pose including a position of a plurality of sports participants, a virtual point 310 corresponding to the image capture device 300, and a line of sight of the image capture device 300. It is a drawing showing virtual lines 320 and 322.

The positions of the plurality of sports participants may be positions obtained from image data or positioning sensor data. According to a preferred example, the positions L2_sensor and L3_sensor of each of the plurality of sports participants may be positions obtained from positioning sensor data. Also, preferably, the positions of the plurality of sports participants may be positions obtained by coordinate conversion of data related to positions included in positioning sensor data.

The virtual point 310 corresponding to the image capture device may be a location obtained from arrangement information including location information of the image capture device 300 located around the playfield. According to a preferred example, the virtual point 310 corresponding to the image capture device may be a location obtained by converting data related to the location of the image capture device 300 into coordinates.

Information about the pose of the image capture device related to the virtual line 320 and information about the line of sight of the image capture device related to the virtual line 322 are ) can be obtained from According to a preferred example, the virtual line 322 corresponding to the line of sight of the image capture device corresponds to the positions (L2_sensor, L3_sensor) of the plurality of sports participants or the virtual point 310. It may be a virtual line generated by coordinate transformation with respect to a common coordinate system.

At this time, the player tracking server 1000 considers the virtual line 322 corresponding to the line of sight and the respective positions (L2_sensor, L3_sensor) of the plurality of sports participants, and between the plurality of sports participants. It is possible to detect or determine whether occlusion has occurred. For example, the position L2_senosr of the first sports participant may be obtained from positioning sensor data obtained from the positioning sensor devices 210 and 220 of the first sports participant, and the location L3_senosr of the second sports participant 2 may be obtained from the positioning sensor data obtained from the positioning sensor devices 210 and 220 of the sports participant. At this time, the player tracking server 1000, when the position of the first sports participant (L2_senosr) and the position of the second sports participant (L3_senosr) are substantially located on a straight line with respect to the line of sight, It may be determined or determined that an occlusion event between the first sports participant and the second sports participant has occurred.

As another example, the player tracking server 1000 may be implemented to determine whether an occlusion event has occurred by considering whether the teams of players related to the occlusion event are identical. Specifically, in image data, particularly in the form of an RGB map, teams of players can be distinguished based on the similarity of pixels related to uniforms. In addition, since the positioning sensor data may include an identifier of an individual player, it is possible to distinguish a team among players based on the identifier.

In this case, the player tracking server 1000 may perform an operation of determining that an occlusion event has occurred if the teams of the players related to the occlusion event are the same. On the other hand, if the teams of the players related to the occlusion event are not the same, the player tracking server 1000 may determine that the occlusion event has not occurred.

However, this is just an example, and even if the teams of players related to the occlusion event are not the same, it can be determined that an occlusion event has occurred, and the severity of the occlusion event is calculated from image data in the operation of determining. This could further evaluate the effectiveness related to the positioning of the players.

As another example, the player tracking server 1000 may detect or determine whether an occlusion event occurs in consideration of a situation during a sports game. Specifically, a situation in which a plurality of players are concentrated may occur, such as a corner kick situation during a sports game. At this time, the player tracking server 1000 may recognize such a dense situation and estimate that an occlusion event has occurred.

The above-described occlusion event detection methods are only examples, and the distance between a plurality of players is considered in consideration of the position of the player calculated within an image in the form of an intensity map and an RGB map, or from image data and positioning sensor data, respectively. It should be appreciated that any suitable method for detecting an occlusion event of may be provided in the player tracking server 1000.

As described above, the player tracking server 1000 according to an embodiment of the present application can detect whether an occlusion event occurs between a plurality of players in image data, and the player tracking server 1000 If it is determined that an occlusion event has occurred, it may be determined that the player's position calculated from the image data is invalid.

The player tracking server 1000 according to an embodiment of the present application further considers the severity of an occlusion event between a plurality of players in the image data to evaluate the validity of the player's position calculated from the image data. It can be. Here, the severity of an occlusion event may be used to encompass a concept that quantifies or quantifies the severity of an occlusion event that at least partially overlaps among a plurality of players.

As an example, referring to FIG. 8 again, the player tracking server 1000 according to an embodiment of the present application includes sports participants located in an area R within a predetermined radius based on the specific location L1_sensor. The severity of the occlusion event may be determined or judged by considering the number, and through this, the effectiveness of the player's position calculated from the image data may be evaluated.

For example, a plurality of threshold values (a first threshold value and a second threshold value) may be preset in relation to the number of sports participants located in the area R within a predetermined radius, and the player tracking server 1000 ) indicates that an occlusion event occurs when the number of sports participants located in the area R within a predetermined radius based on the specific location L1_sensor is greater than the first threshold value and less than the second threshold value. However, the severity can be judged to be mild.

On the other hand, in the player tracking server 1000, when the number of sports participants located in the area R within a predetermined radius based on the specific location L1_sensor is greater than the second threshold, an occlusion event has occurred and an occlusion event has occurred. It may be determined that the severity of the closure event is severe.

Regardless of the severity of the occlusion event, the player tracking server 1000 determines that the number of sports participants located in the area R within a predetermined radius based on the specific location L1_sensor is less than the first threshold value. In this case, it may be determined that no occlusion event has occurred.

As another example, even if the player tracking server 1000 according to an embodiment of the present application determines that an occlusion event has occurred, the validity of the player's position calculated from image data by additionally considering a team identifier. can be provided to evaluate. For example, even if an occlusion event among a plurality of sports participants included in image data occurs, the player tracking server 1000 performs occlusion according to team identifiers of players related to the occlusion event. By additionally determining the severity of the event, the validity of the player's position calculated from the image data can be evaluated.

Generally, when sports participants are on the same team, their uniforms are different. Therefore, if an occlusion event occurs between sports participants belonging to the same team, there is a high possibility that the location of sports participants calculated from image data will not be valid.

On the other hand, if the teams of sports participants are not the same, uniforms are different, so even if an occlusion event occurs between sports participants belonging to different teams, pixel data of the RGB map related to the color of the uniform is used. The location of the participants can be obtained effectively.

Therefore, even if it is determined that an occlusion event has occurred, the player tracking server 1000 according to an embodiment of the present application considers the team identifier additionally to determine the validity of the player's position calculated from the image data. that can be evaluated.

As the team identifier, pixel data related to the color of the uniform obtained from the image data has been exemplified, but this is only an example, and may be implemented by further including the team identifier as well as the player identifier in the positioning sensor data. There will be.

See Figure 9. 9(a) and 9(b) are exemplary diagrams of a method for determining the validity of a player's position calculated from image data disclosed in the present application.

For example, the player tracking server 1000 may be provided to evaluate the validity of the player's position calculated from the image data in consideration of the player's vertical movement included in the image data. Here, vertical movement may mean encompassing all movements in a vertical direction from a coordinate system plane related to a player's position calculated from image data. Specifically, data related to the vertical movement may be further obtained from an inertial sensor worn on the players, and if there is a vertical movement, an error may occur in the position of the player calculated from the image data. Possibilities exist. Accordingly, the player tracking server 1000 may evaluate the validity of the player's position calculated from the image data in consideration of the player's vertical movement included in the image data.

Referring to FIG. 9(a) , when there is substantially no vertical movement of the sports participant 1 included in the image data, the position (L1_image) of the sports participant 1 calculated from the image data and A difference in the actual position L1 of the sports participant 1 may be relatively small. In other words, when there is substantially no vertical movement of the sports participant 1 included in the image data, the position L1_image of the sports participant 1 calculated from the image data is relative to the actual position L1. can be calculated as an approximation.

On the other hand, referring to FIG. 9(b), when there is a vertical movement of the sports participant 1 included in the image data, the position (L2_image) of the sports participant 1 calculated from the image data A difference between the actual position L2 of the sports participant 1 may be relatively large. In other words, if there is a vertical movement of the sports participant 1 included in the image data, the position (L2_image) of the sports participant 1 calculated from the image data is significant with respect to the actual position (L2). error is likely to occur.

Accordingly, the player tracking server 1000 may evaluate the validity of the player's position calculated from the image data in consideration of the player's vertical movement included in the image data. For example, the player tracking server 1000 detects the vertical movement of the player included in the image data, that is, the change value of the z-direction data from the inertial sensor worn by the sports participant 1 If detected, it may be determined or judged that the position of the sports participant calculated from the image data is not valid.

As another example, the player tracking server 1000 may perform a change in z-direction data from an inertial sensor worn by the sports participant 1 when the vertical movement of the player included in the image data is not detected. If no value is detected, the position of the sports participant 1 calculated from the image data can be determined or judged to be valid.

As described above, it has been described that the validity of the sports participant calculated from the image data can be evaluated according to whether the vertical movement of the sports participant is detected, but this is only an example and vertical movement Even if it is detected, it may be implemented to further consider the direction or magnitude of the acceleration in the z-direction to determine the validity of the sports participant's position calculated from the image data. In addition, information about the height, position, and capture angle of the image capture device 300 may be additionally provided to be considered in determining whether the position of the sports participant calculated from the image data is valid.

See FIG. 10 . 10 is an exemplary diagram of a method for determining the validity of a player's position calculated from image data disclosed in the present application.

For example, the player tracking server 1000 may evaluate the validity of the player's position calculated from the image data in consideration of the internal disparity of the player's position calculated from the image data. Specifically, the player tracking server 1000 may continuously calculate the player's position from the image data. At this time, if the calculated player's position changes more than a predetermined threshold value, the player tracking server 1000 Data on the player's position at the point of time when it significantly changed can be determined to be invalid. In other words, when the change in the player's position calculated from the image data corresponding to the first viewpoint and the player's position calculated from the image data corresponding to the second viewpoint change more than a predetermined threshold value, the player tracking server ( 1000) may determine that the position of the player calculated from the image data corresponding to the second viewpoint is invalid.

Referring to FIG. 10 , the player tracking server 1000 may be configured to calculate or obtain positions of a particular sports participant over time from image data. At this time, when the difference between the position of the sports participant (Lt1_image) calculated from the first viewpoint and the position (Lt2_image) of the sports participant calculated from the second viewpoint is greater than a predetermined threshold, the player tracking server 1000 determines the second It may be determined that the location (Lt2_image) of the sports participant calculated from the viewpoint is invalid. On the other hand, when the difference between the position of the sports participant (Lt3_image) calculated from the third viewpoint and the position (Lt4_image) of the sports participant calculated from the fourth viewpoint is smaller than a predetermined threshold value, the player tracking server 1000 determines the fourth The position (Lt4_image) of the sports participant calculated from the viewpoint may be determined to be valid.

According to the foregoing, it has been described that the validity of the position over time of a sports participant calculated from image data based on a preset threshold is determined, but this is only an example, and any suitable method can be used to determine the validity of the location from image data. The validity of the calculated position of the sports participant can be judged. For example, the player tracking server 1000 may calculate a trend line by quantifying the tendency of positions of the sports participant over time, and the sports participant calculated from the image data based on the magnitude of the deviation from the trend line. will be able to judge the validity of the location of

In addition, the player tracking system 100 according to an embodiment of the present application may determine the validity of the obtained positioning sensor-based position. Specifically, the player tracking server 1000 may evaluate the validity of the player's position calculated from positioning sensor data. More specifically, the player tracking server 1000 evaluates the validity of the player's position calculated from the positioning sensor data, information on the reliability of the positioning sensor data, the distance of the player's position calculated from the positioning sensor data (internal disparity) may be considered.

See Figure 11. 11 is an exemplary diagram of a method for determining the validity of a player's position calculated from positioning sensor data disclosed in the present application.

For example, the player tracking server 1000 may evaluate the validity of the player's position calculated from the positioning sensor data in consideration of information related to the reliability of the sensor signal included in the positioning sensor data.

For example, a signal transmitted from a global navigation satellite system (GNSS) to a positioning sensor device may include information related to signal reliability (eg, DoP, SNR).

At this time, the information related to the reliability may be acquired by the player tracking server 1000 through the communication module 1100, and the player tracking server 1000 may position the player in consideration of the obtained information related to the sensor-signal reliability. The validity of the player's position calculated from the sensor data can be evaluated.

For example, a threshold may be set in advance in relation to the information related to the reliability, and when the information related to the reliability included in the positioning sensor data is greater than the preset threshold, the player tracking server 1000 It may be determined that the position of the player calculated from the positioning sensor data is valid. On the other hand, if information related to reliability included in the positioning sensor data is smaller than the preset threshold, the player tracking server 1000 may determine that the position of the player calculated from the positioning sensor data is invalid.

Referring to FIG. 11 , information related to the reliability of the sensor-signal may be different for each area of the playfield. For example, when a sports participant is located in the first area A1 of the playfield, reliability of the sensor-signal may be relatively low. Therefore, location data included in positioning sensor data obtained from the positioning sensor device of a sports participant located in the first area A1 may have relatively low reliability. On the other hand, when a sports participant is located in the second area A2 of the playfield, the reliability of the sensor-signal may be relatively high. Accordingly, position data included in positioning sensor data acquired from the positioning sensor device of a sports participant located in the second area A2 may have relatively high reliability.

In this case, when the reliability of the sensor-signal in the first area A1 is higher than a predetermined threshold value, the player tracking server 1000 obtains a positioning sensor device for a sports participant located in the first area A1. The position of the sports participant calculated from the acquired positioning sensor data may be determined or determined as valid. On the other hand, when the reliability of the sensor-signal in the second area A2 is lower than the predetermined threshold value, the player tracking server 1000 obtains a positioning sensor device for a sports participant located in the second area A2. The position of the sports participant calculated from the acquired positioning sensor data may be judged or determined to be invalid.

In addition, the player tracking server 1000 quantifies and compares the reliability of the sensor-signal and the reliability of the image data to determine that the location of the sports participant calculated based on highly reliable data is valid, or the sensor-signal reliability and image data The position of the sports participant can be calculated through weights according to the reliability of This will be described later with reference to FIG. 22 .

In FIG. 11, the first area and the second area are limited to specific areas within the playfield, but this is only one example for explaining that the sensor-signal reliability is different for each area.

See Figure 12. 12 is an exemplary diagram of a method for determining the validity of a player's position calculated from positioning sensor data disclosed in the present application.

For example, the player tracking server 1000 may evaluate the effectiveness of the player's position calculated from the positioning sensor data in consideration of the internal disparity of the player's position calculated from the positioning sensor data. Specifically, the player tracking server 1000 may continuously calculate or obtain the player's position from positioning sensor data, and in this case, if the calculated or acquired player's position changes more than a predetermined threshold value, The player tracking server 1000 may determine that data on the position of the player at the point of time that greatly changes is invalid. In other words, when the change in the position of the player calculated from the positioning sensor data corresponding to the first viewpoint and the position of the player calculated from the positioning sensor data corresponding to the second viewpoint change more than a predetermined threshold value, the player is tracked. The server 1000 may determine that the player's position calculated from positioning sensor data corresponding to the second viewpoint is invalid.

Referring to FIG. 12 , the player tracking server 1000 may calculate or obtain positions of specific sports participants over time from positioning sensor data. At this time, when the difference between the sports participant's position (Lt1_sensor) calculated from the first time point and the sports participant's location (Lt2_sensor) calculated from the second time point is greater than a predetermined threshold value, the player tracking server 1000 The sports participant's position (Lt2_sensor) calculated from two viewpoints may be determined to be invalid. On the other hand, when the difference between the sports participant's position (Lt3_sensor) calculated from the third time point and the sports participant's location (Lt4_sensor) calculated from the fourth time point is smaller than a predetermined threshold value, the player tracking server 1000 may determine that the sports participant's position (Lt4_sensor) calculated from the fourth viewpoint is valid.

According to the above, it has been described that the validity of the position over time of the sports participant calculated from the positioning sensor data based on the preset threshold is determined, but this is only an example, and any appropriate method can be used to determine the positioning sensor. The validity of the sports participant's position calculated from the data can be determined. For example, the player tracking server 1000 may calculate a trend line by quantifying the tendency of positions of a sports participant over time, and calculate a sport calculated from positioning sensor data based on a size of deviation from the trend line. It will be possible to judge the validity of the participant's location.

The player tracking system 100 according to an embodiment of the present application may determine validity of at least one of the image-based position and the positioning sensor-based position based on a difference between the image-based position and the positioning sensor-based position. Specifically, the player tracking server 1000 may be provided to determine the validity of at least one of an image-based position and a positioning sensor-based position. More specifically, the player tracking server 1000 may be provided to determine validity of at least one of the image-based position and the positioning sensor-based position, based on an extensive disparity between the image-based position and the positioning sensor-based position. there is.

For example, the player tracking server 1000 may continuously calculate or obtain the player's position from positioning sensor data and image data. At this time, based on a disparity index considering the difference between the first position of the player calculated from the image data and the second position of the player calculated from the positioning sensor data (hereinafter referred to as external disparity) for the same temporal point in time, the The player tracking server 1000 may evaluate the validity of at least one of the player's position calculated from image data and the player's position calculated from positioning sensor data.

For example, the distance index between the first position and the second position calculated at a first time point may be a first distance index, and the first position and the second time point calculated at a second time point temporally adjacent to the first time point. The distance index between the second positions may be a second distance index.

)의 절대값이 미리 정해진 제1 임계값 미만인 경우에는, 상기 선수 추적 서버(1000)는 제1 시점의 이미지 데이터로부터 계산된 선수의 제1 위치와 제1 시점의 측위 센서 데이터로부터 계산된 선수의 제2 위치 중 적어도 하나 이상이 유효한 것으로 판단할 수 있다. At this time, the first separation index (

) is less than the first predetermined threshold value, the player tracking server 1000 calculates the first position of the player from the image data of the first view and the positioning sensor data of the first view of the player. At least one of the second positions may be determined to be valid.

)의 절대값이 미리 정해진 제2 임계값을 초과하는 경우에는, 상기 선수 추적 서버(1000)는 제2 시점의 이미지 데이터로부터 계산된 선수의 제1 위치와 제2 시점의 측위 센서 데이터로부터 계산된 선수의 제2 위치 중 적어도 하나가 유효하지 않은 것으로 평가할 수 있다. On the other hand, the second separation index (

) exceeds a predetermined second threshold, the player tracking server 1000 calculates the first position of the player calculated from the image data of the second viewpoint and the positioning sensor data of the second viewpoint. At least one of the athlete's second positions may be evaluated as invalid.

Here, the first threshold value and the second threshold value may be the same as each other, but may be preset to any suitable value different from each other.

)과 상기 제2 이격도 지수(

)의 차이(

) 혹은

)를 고려하여, 상기 선수 추적 서버(1000)는 이미지 데이터로부터 계산된 선수의 위치와 측위 센서 데이터로부터 계산된 선수의 위치 중 적어도 하나의 유효성을 평가할 수 있다. In another example, the first separation index (

) and the second separation index (

) of the difference (

) or

), the player tracking server 1000 may evaluate the validity of at least one of the player's position calculated from image data and the player's position calculated from positioning sensor data.

)와 상기 제2 이격도 지수(

)의 차이(

) 혹은

)가 미리 결정된 제3 임계값 미만인 경우(바람직하게는, 제1 이격도 지수과 관련된 제1 위치와 제2 위치는 유효하다고 판단된 경우), 즉 제2 시점의 이미지 데이터로부터 계산된 선수의 제1 위치와 제2 시점의 측위 센서 데이터로부터 계산된 선수의 제2 위치 간의 차이가 오차 범위 내라고 볼 수 있는 경우에는, 상기 선수 추적 서버(1000)는 제2 시점의 제1 위치와 제2 위치 모두가 유효한 것으로 판단할 수 있다.Specifically, the first separation index (

) and the second separation index (

) of the difference (

) or

) is less than a predetermined third threshold (preferably, when it is determined that the first position and the second position related to the first separation index are valid), that is, the first position of the player calculated from the image data of the second viewpoint If the difference between the first position and the second position of the player calculated from the positioning sensor data at the second viewpoint is within the error range, the player tracking server 1000 determines that both the first position and the second position at the second viewpoint are valid. can be judged to be

)과 상기 제2 이격도 지수 (

)의 차이(

) 혹은

)가 미리 결정된 제4 임계값보다 초과하는 경우(바람직하게는, 제1 이격도 지수과 관련된 제1 위치와 제2 위치는 유효하다고 판단된 경우), 즉 제2 시점의 이미지 데이터로부터 계산된 선수의 제1 위치와 제2 시점의 측위 센서 데이터로부터 계산된 선수의 제2 위치 간의 차이가 크게 변하는 시점이 검출된 경우에는, 상기 선수 추적 서버(1000)는 제2 시점의 제1 위치와 제2 위치 중 적어도 하나의 위치가 유효하지 않다고 판단할 수 있다. On the other hand, the first separation index (

) and the second separation index (

) of the difference (

) or

) exceeds a predetermined fourth threshold (preferably, when it is determined that the first position and the second position related to the first separation index are valid), that is, the player's second position calculated from the image data of the second viewpoint. When a time point at which the difference between the first position and the second position of the player calculated from the positioning sensor data at the second time point greatly changes is detected, the player tracking server 1000 determines whether the first position and the second position at the second time point are It may be determined that at least one location is not valid.

Here, the third threshold value and the fourth threshold value may be the same as each other, but may be preset to any suitable value different from each other.

The player tracking system 100 according to an embodiment of the present application may be provided to generate a confidence index related to image-based location and positioning sensor-based location. In detail, the player tracking server 1000 may generate a reliability index related to the image-based location and a reliability index related to the positioning sensor-based location, based on a result of determining validity of the image-based location and the positioning sensor-based location.

In detail, the player tracking server 1000 may generate a first reliability index of the player's position calculated from the positioning sensor data based on a result of evaluating the validity of the positioning sensor data. For example, the first reliability index may be quantified from information related to reliability of a signal included in the positioning sensor data.

Also, the player tracking server 1000 may generate a second reliability index of the player's position calculated from the image data based on a result of evaluating the validity of the image data. In this case, the second reliability index may be quantified as an index corresponding to the first reliability index for quantitative comparison with the first reliability index.

At this time, the generated first reliability index and the second reliability index may be used as one of the consideration factors in finally determining the position of a sports participant, which will be described later.

The player tracking system 100 according to an embodiment of the present application may be provided to convert coordinates between an image-based position and a positioning sensor-based position. Specifically, the player tracking server 1000 may perform coordinate conversion between an image-based position and a positioning sensor-based position. Specifically, the player tracking server 1000 may be provided to perform coordinate conversion between an image-based position and a positioning sensor-based position. Through this, coordinates between the image-based position and the positioning sensor-based position may be unified.

Specifically, the player's position calculated from image data may be calculated with respect to a first coordinate system, and the player's position calculated from positioning sensor data may be calculated with respect to a second coordinate system. At this time, as necessary for tracking the player, the player's position is determined by considering both the player's position calculated from the image data and the player's position calculated from positioning sensor data, or the player's position and In the case of "switching" between positions of the player calculated from positioning sensor data, a conversion operation between the first coordinate system and the second coordinate system must be performed because the player's position must be determined with respect to a common coordinate system.

For example, the second coordinate system may be transformed with respect to the first coordinate system.

As another example, the first coordinate system may be transformed with respect to the second coordinate system.

As another example, the calculated position of the player may be converted with respect to the first coordinate system and the second coordinate system with respect to an absolute coordinate system different from the first coordinate system and the second coordinate system.

However, the conversion of the above-described coordinate system is only an example, and the position of the player calculated from the image data and the position of the player calculated from positioning sensor data can be variously converted with respect to the common coordinate system according to any suitable method.

At this time, the player tracking server 1000 may be provided to use various coordinate conversion algorithms.

For example, the player tracking server 1000 may be implemented using an artificial neural network trained by any suitable method such as machine learning or deep learning.

See Figures 13 to 14.

13 is a diagram illustrating training of an artificial neural network for converting a position of a first coordinate system related to positioning sensor data into a position of a second coordinate system related to image data.

See Figure 14. 14 is a diagram illustrating conversion of a position of a first coordinate system related to positioning sensor data into a position of a second coordinate system related to image data using an artificial neural network.

Referring to FIG. 14 , the player tracking server 1000 according to an embodiment of the present application converts the position of a first coordinate system related to positioning sensor data into a position of a second coordinate system related to image data using an artificial neural network. can be performed.

For example, when it is determined that the image-based location obtained from the image data is not valid, there is a need to determine the location of the sports participant from the location sensor-based location obtained from the location sensor data. However, since the positioning sensor-based position and the image-based position may have different coordinate systems, the player tracking server 1000 may convert the obtained positioning sensor-based position of the first coordinate system to a position of the second coordinate system related to the image data. there is. Here, when the image-based position is not valid, occlusion between sports participants is detected, when the occlusion event is determined to be serious, when a vertical movement of the sports participant is detected, and the distance between image-based positions (internal disparity ) may mean at least one of the cases in which there exists.

For example, an artificial neural network to be learned to convert a position of a first coordinate system related to positioning sensor data to a position of a second coordinate system related to image data may include an input layer, a hidden layer including a plurality of nodes, and an output layer. can

In order to train the artificial neural network, image-based positions (x_image, y_image) determined to be valid and positioning sensor-based positions (x_sensor, y_sensor) corresponding to the image-based positions determined to be valid may be input to the input layer. In addition, for accurate learning, positioning sensor-based positions (x_sensor, y_sensor) and positioning sensor-based speed (|v|_sensor) corresponding to the image-based position determined to be valid may be additionally input to the input layer. In this case, the positioning sensor-based speed (|v|_sensor) may be calculated from the positioning sensor-based position (x_sensor, y_sensor). Alternatively, the positioning sensor-based speed (|v|_sensor) may be calculated from data obtained from an accelerometer of an inertial sensor worn by an athlete.

For example, referring to FIG. 13 , image data includes image data in which occlusion is not detected (Image 1), occlusion detected but not severe (Image 2), and occlusion detected and severe. may include image data (image 3) determined to be

At this time, according to the above description, the image-based position obtained by the image data (image 1) in which occlusion is not detected may be valid. In other words, it can be used to learn artificial neural networks. Accordingly, the first image-based position (x_image1, y_image1) obtained by the image data (image 1) in which occlusion is not detected and the first positioning sensor-based position (x_sensor1, y_sensor1) corresponding to the first image-based position and the speed based on the first positioning sensor (|v|_sensor1) may be used as an input value of the learning set.

At this time, according to the foregoing, the occlusion has been detected and the image-based location obtained by the image data (image 3) determined to be serious may not be valid. In other words, if it is used to learn an artificial neural network, the accuracy of learning may decrease. Accordingly, data related to the image data (image 3) in which the occlusion has been detected and determined to be serious may not be used as a training set.

In addition, image data (Image 2) in which occlusion is detected but determined not to be serious may be appropriately used or not used as a training set in consideration of the above-described criteria.

Referring to FIG. 13 again, the positioning sensor-based position (x_sensor, y_sensor) and the positioning sensor-based speed (|v|_sensor) corresponding to the image-based position determined to be valid input to the input layer pass through the hidden layer to the output layer. As an output value, it can be converted into a position (x'_image, y'_image) relative to the coordinates of the image-based position.

At this time, the output values (x'_image, y'_image) may be compared with the image-based positions (x_image, y_image) input to the input layer. Through this, the artificial neural network can be learned by adjusting the weight of the node included in the hidden layer based on the difference between the output value and the image-based location.

Referring to FIG. 14 , the position of the first coordinate system related to positioning sensor data may be converted into the position of the second coordinate system related to image data using the artificial neural network trained according to FIG. 13 . The trained artificial neural network according to an embodiment of the present application preferably converts the position of the first coordinate system related to the positioning sensor data to the position of the second coordinate system related to the image data when it is determined that the image-based position is not valid. can be used to do

For example, the image data includes image data with no occlusion detected (Image 1), image data with occlusion detected but determined not to be serious (Image 2), and image data with occlusion detected and determined to be serious (Image 2). Image 3) may be included.

At this time, it may be determined that occlusion has been detected among the image data and the image-based positions (x_image3, y_image3) obtained by the image data (image 3) determined to be serious are invalid. In this case, occlusion was detected and positioning sensor-based position (x_sesnor3, y_sensor3) and localization sensor-based speed (|v If |_sensor3) is input as an input layer of the learned artificial neural network, the localization sensor-based position (x_sesnor3, y_sensor3) can be converted into a coordinate system related to the image-based position. Through this, even when the image-based location is not valid, the location of the sports participant may be continuously obtained based on the positioning sensor-based location.

At this time, in relation to the image data (Image 2) in which occlusion is detected but determined not to be serious among the image data, a reliability index between the image-based position and the positioning sensor-based position, which will be described later, is compared to determine the positioning sensor-based position as the image-based position. If the reliability is higher, it may be provided to convert the coordinate values using the learned artificial neural network.

According to the foregoing, although the conversion of the positioning sensor-based position of the first coordinate system to the position of the second coordinate system related to the image-based position has been described, the artificial neural network will be trained to transform the position of any appropriate common coordinate system. You will be able to.

In addition, although the conversion of the coordinates of the positioning sensor-based position when the image-based position is not valid has been described, this is only an example, and when it is determined that the positioning sensor-based position is not valid, by a similar method, It is self-evident that it is possible to learn or deploy artificial neural networks that convert image-based positions to localization sensor-based positions.

As described above, in the present application, the player tracking server 1000 may use various coordinate conversion algorithms for converting coordinates.

For example, the coordinate transformation algorithm may be provided as a machine learning model. A typical example of a machine learning model may be an artificial neural network. Specifically, a representative example of an artificial neural network is a deep learning artificial neural network including an input layer that receives data, an output layer that outputs results, and a hidden layer that processes data between the input layer and the output layer. . Specific examples of the artificial neural network include a convolution neural network, a recurrent neural network, a deep neural network, and the like. It should be interpreted as a comprehensive meaning that includes both artificial neural networks of various types and artificial neural networks of a combination thereof, and does not necessarily have to be a deep learning series.

In addition, the machine learning model does not necessarily have to be in the form of an artificial neural network model, and may include nearest neighbor algorithm (KNN), random forest, support vector machine (SVM), principal component analysis (PCA), etc. The techniques mentioned above may include all of the ensemble forms or even forms combined in various ways. On the other hand, in the embodiments referred to centering on the artificial neural network, it is disclosed in advance that the artificial neural network may be replaced with another machine learning model unless otherwise specified.

Furthermore, the coordinate transformation algorithm in this specification is not necessarily limited to a machine learning model. That is, the coordinate conversion algorithm may include various judgment/determination algorithms other than the machine learning model.

Therefore, it is made clear in advance that the coordinate conversion algorithm in this specification should be understood in a comprehensive sense that includes all types of algorithms that perform coordinate conversion using data on the first coordinate system.

The player tracking system 100 according to an embodiment of the present application may be provided to perform interpolation when converting coordinates between an image-based position and a positioning sensor-based position. Specifically, the player tracking server 1000 interpolates the converted coordinate values in order to correct errors that may occur in the process of converting the coordinates between the player's position calculated from image data and the player's position calculated from positioning sensor data. ) can be additionally performed. For example, if it is determined that the player's position calculated from the image data is not valid while tracking the player with the player's position calculated from the image data, the player is tracked with the player's position calculated from the positioning sensor data as necessary. may have to At this time, when the player's position calculated from the positioning sensor data is converted to correspond to the first coordinate system of the player's position calculated from the image data, an error may exist in the value of the converted position.

Therefore, the player tracking server 1000 according to an embodiment of the present application converts the coordinates of the player's position calculated from the positioning sensor data corresponding to the first coordinate system to the player's position calculated from the previously tracked image data. It is possible to perform an operation of position and interpolation.

The player tracking system 100 according to an embodiment of the present application may determine the player's position by considering the image-based position and the positioning sensor-based position.

In detail, the player tracking server 1000 may be provided to finally determine the player's position based on the image-based position and the positioning sensor-based position.

For example, if the image-based location is valid, the player tracking server 1000 may determine the player's position based on the player's position calculated from image data.

On the other hand, if the image-based location is not valid, the player tracking server 1000 may determine the player's location based on the location calculated from positioning sensor data.

For example, when it is determined that there is no occlusion event between a plurality of players included in the image data or is not detected, and thus the position of the player calculated from the image data is determined to be valid, the player tracking server 1000 performs the image data The position of the player can be determined based on the position of the player calculated from On the other hand, when it is determined that an occlusion event exists or is detected between a plurality of players included in the image data and the position of the player calculated from the image data is not valid, the player tracking server 1000 determines from the positioning sensor data Based on the calculated position, the position of the player can be determined.

As another example, when it is determined that the player's position calculated from the image data is valid because vertical motion is not detected with respect to the player included in the image data, the player tracking server 1000 determines the player's position calculated from the image data. Based on this, the player's position can be determined. On the other hand, if a vertical motion is detected with respect to the player included in the image data and it is determined that the player's position calculated from the image data is not valid, the player tracking server 1000 based on the position calculated from the positioning sensor data Thus, the position of the player can be determined.

As another example, when it is determined that the player position calculated from the image data is valid because the internal disparity between player positions calculated from the image data does not exceed a predetermined threshold, the player tracking server 1000 The position of the player may be determined based on the position of the player calculated from the image data. On the other hand, when it is determined that the player position calculated from the image data is invalid because the internal disparity between the player positions calculated from the image data exceeds a predetermined threshold value, the player tracking server 1000 uses a positioning sensor The position of the player can be determined based on the position calculated from the data.

As an embodiment, if the player's position calculated from the positioning sensor data is valid, the player tracking server 1000 may determine the player's position based on the player's position calculated from the positioning sensor data. On the other hand, if the player's position calculated from the positioning sensor data is not valid, the player tracking server 1000 may determine the player's position based on the position calculated from image data.

For example, the player tracking server 1000 may determine the player's position based on the determination that the player's position calculated from the positioning sensor data is valid based on information related to the reliability of the signal included in the positioning sensor data. there is.

For example, when it is determined that the player's position calculated from the positioning sensor data is valid because the reliability of a signal included in the positioning sensor data is greater than a preset threshold, the player tracking server 1000 determines from the positioning sensor data The position of the player may be determined based on the calculated position of the player.

On the other hand, when it is determined that the player's position calculated from the positioning sensor data is not valid because the reliability of the signal included in the positioning sensor data is less than a preset threshold, the player tracking server 1000 calculates the position of the player from the image data. Based on the player's position, the position of the player can be determined.

As another example, when it is determined that the player's position calculated from the positioning sensor data is valid because the internal disparity between the player's positions calculated from the positioning sensor data does not exceed a predetermined threshold, the player tracking server ( 1000) may determine the position of the player based on the position of the player calculated from the positioning sensor data.

On the other hand, when it is determined that the player's position calculated from the positioning sensor data is invalid because the internal disparity between the positions of the player calculated from the positioning sensor data exceeds a predetermined threshold, the player tracking server (1000 ) may determine the position of the player based on the position calculated from the image data.

In one embodiment, the player tracking server 1000 may finally determine the player's position based on the player's position difference or separation index calculated from the image data and the positioning sensor data.

)와 제2 시점에서의 상기 이미지 데이터로부터 계산된 선수의 제1 위치와 상기 측위 센서 데이터로부터 계산된 선수의 제2 위치 간의 제2 이격도 지수(

)에 기초하여 상기 제1 위치와 상기 제2 위치 중 적어도 하나가 유효하지 않다고 판단될 수 있다. 이때, 선수 추적 서버(1000)는 상기 이격도 지수에 기초한 유효성 판단에 기초하여 선수의 위치를 최종적으로 결정할 수 있다. Specifically, as described above, the first separation index between the first position of the player calculated from the image data at the first viewpoint and the second position of the player calculated from the positioning sensor data (

) and a second separation index between the first position of the player calculated from the image data at the second viewpoint and the second position of the player calculated from the positioning sensor data (

), it may be determined that at least one of the first location and the second location is invalid. At this time, the player tracking server 1000 may finally determine the position of the player based on the validity judgment based on the degree of separation index.

For example, the player tracking server 1000 determines that if the calculated distance index is smaller than a first predetermined threshold value, that is, both the position of the player calculated from the positioning sensor data and the position of the player calculated from the image data are valid. If it can be determined, the position of the player may be determined based on one of the position of the player calculated from the positioning sensor data and the position of the player calculated from the image data.

On the other hand, if the calculated separation index is greater than the second predetermined threshold, that is, if at least one of the player's position calculated from positioning sensor data and the player's position calculated from image data can be determined to be invalid, The player tracking server 1000 may determine the player's position based on a player's position having a high reliability index among the player's position calculated from the positioning sensor data and the player's position calculated from the image data.

See Figure 15. 15 is a diagram illustrating an exemplary method of determining a player's position based on a confidence index according to an embodiment of the present application.

In an embodiment, the player tracking server 1000 may determine the position of the player based on a comparison between a first reliability index related to the position of the player calculated from image data and a second reliability index related to the position of the player calculated from positioning sensor data. can decide

For example, when the first reliability index is higher than the second reliability index (FIG. 15 (a) or FIG. 15 (b)), the player tracking server 1000 based on the player's position calculated from image data Player position can be determined.

For example, as in the case of FIG. 15(a), when the first reliability index is higher than the predetermined reliability threshold and the second reliability index is lower than the predetermined reliability threshold, the player tracking server 1000 The sports participant's location may be determined based on the sports participant's location calculated from the image data.

For example, as in the case of FIG. 15(b), when the first reliability index is higher than the second reliability index and both the first reliability index and the second reliability index are higher than a predetermined reliability threshold value, , Similar to FIG. 15(a), the player tracking server 1000 may determine the location of the sports participant based on the location of the sports participant calculated from image data.

As another example, when the second reliability index is higher than the first reliability index (in the case of FIG. 15(c)), the player tracking server 1000 based on the position of the player calculated from the positioning sensor data Player position can be determined.

As another example, as in the case of FIG. 15(c), when the second reliability index is higher than the predetermined reliability threshold and the first reliability index is lower than the predetermined reliability threshold, the player tracking server (1000 ) may determine the location of the sports participant based on the location of the sports participant calculated from positioning sensor data.

As another example, the player tracking server 1000 considers the first reliability index related to the first position of the player calculated from the image data and the second reliability index related to the second position of the player calculated from the positioning sensor data and assigns a weight. Thus, the position of the player can be determined. Specifically, the player tracking server 1000 may determine the player's position by assigning a weight considering the first reliability index and the second reliability index to each of the first position and the second position.

For example, as shown in FIG. 15(b), when both the first reliability index and the second reliability index are higher than a predetermined reliability threshold, the player tracking server 1000 determines the sports participant's sports participant's score calculated from the image data. The location of the sports participant may be determined by assigning a weight according to the reliability index to the first location and the second location of the sports participant calculated from positioning sensor data.

As another example, as in the case of FIG. 15(d), when the first reliability index is similar to or substantially the same as the second reliability index, the player tracking server 1000 calculates the sports participant from image data. The position of the sports participant may be determined by assigning a weight according to the reliability index to the position of the sports participant calculated from the location of the position and positioning sensor data.

According to the foregoing, it has been described that the position of a sports participant can be determined by assigning a weight according to the reliability index only in the case of FIGS. 15(b) and 15(d), but this is only an example, and FIG. 15(a) , FIG. 15(c) or in any case, it is obvious that the position of the sports participant can be determined by appropriately assigning a weight according to the reliability index.

The location of the sports participant determined by the player tracking server 1000 according to an embodiment of the present application may be transmitted to the memory 1200 through the communication module 1100 and stored therein. Alternatively, the location of the sports participant determined by the player tracking server 1000 may be transmitted to an external server through the communication module 1100 or uploaded to the Internet.

In the above, the configuration and operation of the player tracking server 1000 according to "one embodiment" of the present application have been described. Hereinafter, the player tracking method according to the present embodiment will be described. In the following description, it will be described that the player tracking method according to an embodiment of the present application is performed by the player tracking system 100 described above. However, since this is only for convenience of description, the player tracking method according to an embodiment of the present application is not limited to the player tracking system 100 described above. That is, the player tracking method to be described later does not necessarily have to be performed only by the player tracking system 100, and may be performed by other systems or devices having functions similar to those of the player tracking system 100 described above.

In addition, the above-described configuration and operation of the player tracking server 1000 according to an embodiment of the present application may be appropriately applied to other embodiments of the player tracking system 100 to be described later.

See Figure 16. 16 is a flowchart illustrating a player tracking method implemented by the player tracking system 100 according to an embodiment of the present application.

A player tracking method according to an embodiment of the present application may include acquiring image data (S1100), acquiring positioning sensor data (S1200), and determining a player's location (S1300).

In the acquiring of the image data ( S1100 ), image data captured from the image capture device 300 located around the playfield may be acquired.

In the acquiring of the positioning sensor data ( S1200 ), positioning sensor data transmitted from the positioning sensor device 200 worn on the sports participant 1 may be obtained.

At this time, the image data and positioning sensor data acquired by the player tracking server 1000 may be data synchronized in time between the image data and the positioning sensor data.

Also, the image data obtained by the player tracking server 1000 may be resampled data.

Also, the image data acquired by the player tracking server 1000 may be noise-removed data.

Also, the positioning sensor data obtained by the player tracking server 1000 may be data that has undergone processing such as noise removal, amplification, and filtering of the positioning sensor data.

Also, the image data acquired by the player tracking server 1000 may include data related to the location of the sports participant and data about a pixel corresponding to the sports participant.

In addition, the player tracking server 1000 may additionally obtain identification data related to the type of the image capture device that has captured the image data and arrangement information related to the location and pose of the captured image capture device. However, the above information is only an example and can be implemented to include any suitable data.

Also, the positioning sensor data obtained by the player tracking server 1000 may include data related to the position of a sports participant, data related to reliability of a sensor-signal, and data related to an identification factor of a sports participant. However, the above information is only an example and can be implemented to include any suitable data.

In the step of determining the position of the player (S1300), the position of the player may be determined based on at least one of the image data and the positioning sensor data.

For example, the player tracking server 1000 verifies the validity of the image-based position calculated from the image data, and if the image-based position is valid, the player's position may be determined based on the image-based position. On the other hand, if the image-based position calculated from the image data is not valid, the position of the player may be determined based on the position of the player calculated from positioning sensor data. This will be described in detail in FIGS. 17 to 19 .

As another example, the player tracking server 1000 may verify the validity of the positioning sensor-based position calculated from the positioning sensor data, and determine the player's position based on the positioning sensor-based position if the positioning sensor-based position is valid. On the other hand, if the positioning sensor-based position calculated from the positioning sensor data is not valid, the player's position may be determined based on the image-based position calculated from image data. Alternatively, if the positioning sensor-based position calculated from the positioning sensor data is not valid, the player tracking server 1000 additionally evaluates the validity of the image-based position and determines the position of the player based on the validity result of the image-based position. can be determined. This will be described in detail in FIGS. 20 to 21 .

As another example, the player tracking server 1000 may determine the position of the player by quantifying and comparing the reliability of the positioning sensor-based position obtained from positioning sensor data and the reliability of the image-based position obtained from image data. Specifically, the player tracking server 1000 determines the player's position based on a position with high reliability among the positioning sensor-based position and the image-based position, or assigns a weight according to the reliability to each of the positioning sensor-based position and the image-based position. Player position can be determined. This will be described later in detail with reference to FIG. 22 .

As another example, the player tracking server 1000 may determine the player's position based on a reliability map of positions based on positioning sensors for each area of the playfield and a reliability map of image-based positions for each area of the playfield. More specifically, the reliability of the player's positioning sensor-based location calculated according to the area of the playfield may be different. In addition, the reliability of the calculated image-based position of the player may be different according to the area of the playfield. Therefore, the player tracking server 1000 determines the player's position by selecting one of the positioning sensor-based position and the image-based position based on the reliability map, or determines the positioning sensor-based position and the image-based position according to the reliability, respectively. Weights can be assigned to determine the player's position. This will be described later in detail with reference to FIGS. 23 to 25 .

Determining the location of a sports participant from image data obtained from an image capture device can accurately measure the location of a sports participant, is relatively less affected by structures around the playfield, and recognizes or analyzes the motion of a sports participant. It is relatively easy to do, and has the advantage of being convenient to use indoors. In addition, since the sports participant does not have to wear the positioning sensor device, it may be convenient from the side of the sports participant. However, an occlusion event may occur between sports participants, and since the capacity of image data is large, a relatively large amount of computation may occur.

On the other hand, determining the position of a sports participant from the positioning sensor data obtained from the positioning sensor device does not generate an occlusion event that may occur in determining the position of a sports participant from image data, and positioning sensor data rather than image data. Since the capacity of is relatively small, processing efficiency of positioning sensor data may be relatively superior to processing efficiency of image data.

Accordingly, an exemplary player tracking method according to an embodiment of the present application determines a sports participant's location from image data, but only when the sports participant's location calculated from the image data is invalid, from positioning sensor data. The location of the participant can be determined.

See Figure 17. FIG. 17 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application, and is a diagram in which the step S1300 of FIG. 17 is specifically subdivided.

Referring to FIG. 17 , the step S1300 according to an embodiment of the present application includes detecting occlusion based on image data (S1310), determining the severity of occlusion (S1320), and determining the severity of occlusion (S1320). Determining whether is equal to or greater than a predetermined threshold (S1330), determining the player's position based on positioning sensor data or image data (S1340 or S1350), and outputting the player's position to the determined position (S1360). ) may be included.

In step S1310, an occlusion event between sports participants included in the image data may be detected. Specifically, the above-described player tracking server 1000 may detect an occlusion event between sports participants included in image data. Alternatively, the player tracking server 1000 may determine or determine whether an occlusion event occurs between sports participants included in the image data.

At this time, when it is determined that an occlusion event between sports participants is not detected or an occlusion event between sports players does not occur in step S1310, the player tracking server 1000 based on the position calculated from the image data. This can determine the location of the sports participant.

On the other hand, if an occlusion event between sports participants is detected in step S1310 or it is determined that an occlusion event between sports players has occurred, a step of determining the severity of the occlusion event (S1320) may be performed. there is. In step S1320, the player tracking server 1000 may additionally determine the severity of the detected occlusion event.

For example, the player tracking server 1000 may determine the severity of the occlusion event by considering the degree of overlap between bounding boxes corresponding to sports participants obtained from the image data as described above.

As another example, the player tracking server 1000 may determine the severity of the occlusion event in consideration of a change in the total number of pixels corresponding to sports participants obtained from the image data as described above.

As another example, the player tracking server 1000 may perform an occlusion event based on the number of sports participants located within a predetermined area from a specific sports participant in consideration of the locations of the sports participants obtained from positioning sensor data as described above. severity can be judged.

As another example, the player tracking server 1000 may determine the severity of the occlusion event by considering team information of sports participants related to the occlusion event as described above. Team information of sports participants may be obtained from image data, particularly pixel information corresponding to uniforms of sports participants of an RGB map. Alternatively, team information of sports participants may be obtained from an identification factor included in positioning sensor data.

As another example, the player tracking server 1000 may determine the severity of the occlusion event in consideration of the "circumstance" of the game in which the occlusion event is detected as described above.

At this time, the severity of the occlusion event may be quantified by any suitable method. Alternatively, the occlusion event may be classified according to a degree of severity by any suitable criterion.

In the step S1330, it may be determined whether the result of determining the severity of the occlusion event derived in the step S1320 is equal to or greater than a predetermined value. Specifically, the player tracking server 1000 may determine whether the severity value of the occlusion event quantified in step S1320 is greater than or equal to a predetermined value.

For example, if the severity value of the quantified occlusion event is greater than or equal to a predetermined value, the location of the sports participant calculated based on the image data may be determined to be invalid or predicted. Therefore, if the severity value of the quantified occlusion event is greater than or equal to the predetermined value, in step S1350, the player tracking server 1000 determines the positioning sensor-based position of the sports participant based on positioning sensor data rather than image data. location can be determined.

As another example, if the severity value of the quantified occlusion event is less than the predetermined value, the location of the sports participant calculated based on the image data may be valid or predicted to be valid. Accordingly, if the severity value of the quantified occlusion event is less than the predetermined value, in step S1340, the player tracking server 1000 may determine the location of the sports participant based on the image-based location calculated based on the image data.

Alternatively, the player tracking server 1000 determines whether to determine the player's position based on positioning sensor data or image data according to the classification result of the severity of the occlusion event in step S1320. can decide

For example, the player tracking server 1000 may assign the occlusion event to a mild occlusion event or a serious occlusion event according to the severity of the occlusion event. In this case, the player tracking server 1000 may determine the position of the sports participant based on the image data according to a result of allocating the occlusion event to the mild occlusion event. On the other hand, the player tracking server 1000 may determine the position of the sports participant based on positioning sensor data according to a result of allocating an occlusion event as a severe occlusion event.

In step S1360, the location of the sports participant may be output or updated based on the determined location of the sports participant. The location of the sports participant to be outputted may be a coordinate value corresponding to any suitable coordinate system.

See Figure 18. 18 is a flowchart illustrating a learning process of an artificial neural network related to a coordinate conversion algorithm according to an embodiment of the present application.

The learning method using the artificial neural network of the coordinate conversion unit according to an embodiment of the present application includes acquiring image data and image-based location (S2100), acquiring sensor data and positioning sensor-based location (S2200), image-based It may include verifying the location (S2300), preparing a learning set (S2400), and training an artificial neural network using the learning set (S2500). The learning method using the present artificial neural network may be implemented by the player tracking server 1000 according to an embodiment of the present application or may be implemented in any separate learning module.

In step S2100, image data transmitted from the image capture device 300 may be acquired. Also, an image-based location may be obtained from the image data. The image-based location may be obtained from a coordinate value of a pixel corresponding to a location of a sports participant included in image data.

In step S2200, positioning sensor data transmitted from the positioning sensor devices 210 and 220 may be obtained. Also, a positioning sensor-based position may be obtained from the positioning sensor data. The positioning sensor-based position may be obtained from position-related data included in positioning sensor data, or may be obtained from speed or acceleration-related data included in positioning sensor data. Specifically, in step S2200, positioning sensor-based positions and positioning sensor-based speeds corresponding to image-based positions may be acquired.

In step S2300 , it may be verified whether the image-based location acquired in step S2100 can be used as a learning set based on a result of determining validity of the image-based location.

For example, the player tracking server 1000 may determine the validity of the image-based position based on whether an occlusion event is detected. In other words, in step S2300 , it may be verified whether the image-based position can be used as a learning set based on the detection result of the occlusion event in the image data.

For example, if an occlusion event in the image data is detected, the obtained image-based location may be determined to be inappropriate as a training set.

For example, if an occlusion event in the image data is not detected (or an occlusion event is detected but determined not to be serious), it is determined that the obtained image-based position can be used as a training set. can

In step S2400, a learning set and an artificial neural network may be prepared according to the verification result of step S2300.

The artificial neural network may include an input layer, an output layer, and a hidden layer including a plurality of nodes.

The training set may include the image-based location verified in step S2300, a positioning sensor-based position corresponding to the image-based position, and a positioning sensor-based speed. In this case, the image-based location may be a location related to the first coordinate system. Also, the positioning sensor-based location may be a second coordinate system different from the first coordinate system.

In the step S2500, the artificial neural network may be trained using the training set prepared in the step S2400.

For example, a position based on a positioning sensor and a speed based on a positioning sensor among the learning set may be input as an input layer. In this case, the input positioning sensor-based position and positioning sensor-based speed may be converted from the second coordinate system to the first coordinate system through an artificial neural network and output through an output layer. In this case, weights of a plurality of nodes included in the hidden layer may be adjusted based on a difference between an output value output through the output layer and an image-based location of the learning set.

By appropriately adjusting the weights of the plurality of nodes, the artificial neural network can be trained to reduce an error between the image-based location of the training set and output values output through the output layer.

19 is a flowchart illustrating an embodiment of a method for tracking a player through coordinate conversion using the learned artificial neural network of FIG. 18;

A player tracking method through coordinate transformation using a learned artificial neural network includes obtaining image data and image-based location (S3100), acquiring positioning sensor data (S3200), and determining the validity of the image-based location (S3100). S3300) and determining the location of the sports participant (S3400, S3500, S3600).

In step S3100, image data transmitted from the image capture device 300 may be obtained. Also, an image-based location may be obtained from the image data. The image-based location may be obtained from a coordinate value of a pixel corresponding to a location of a sports participant included in image data.

In the step S3200, positioning sensor data transmitted from the positioning sensor device 200 may be acquired.

In step S3300, the validity of the image-based location may be determined by the player tracking server 1000.

For example, the validity of the image-based location obtained from step S3100 may be determined or predicted according to any suitable method for determining validity of an image-based location described above with reference to FIGS. 5 to 10 .

For example, the player tracking server 1000 considers at least one of whether an occlusion event is detected, a result of determining the severity of the occlusion event, a vertical movement of a sports participant, and an internal disparity of an image-based position. The validity of the image-based location can be determined by

In step S3400, if the image-based location is determined to be valid in step S3300, the player tracking server 1000 may determine the location of the sports participant based on the image-based location.

On the other hand, if it is determined that the image-based location is not valid in step S3300, the player tracking server 1000 may determine the location of the sports participant based on positioning sensor data.

In the step S3500, a positioning sensor-based position may be obtained from the positioning sensor data obtained in the step S3200. The positioning sensor-based position may be obtained from position-related data included in positioning sensor data, or may be obtained from speed or acceleration-related data included in positioning sensor data. Specifically, in step S3400, positioning sensor-based positions and positioning sensor-based velocities corresponding to the image-based position may be acquired.

According to FIG. 19, it is shown that the positioning sensor-based position is acquired only when it is determined that the image-based position is not valid in step S3300, but this is only an example, and the positioning sensor in step S3200 as well as in step S3500. It is obvious that positioning sensor based position or positioning sensor based speed can be calculated from the data.

In the step S3600, the positioning sensor-based position and the positioning sensor-based speed acquired in the step S3500 may be input as input values of the learned input layer of the artificial neural network.

The positioning sensor-based position and the positioning sensor-based speed may be values obtained in a second coordinate system. Also, the positioning sensor-based position and the sensor-based speed may be values for a position and speed corresponding to an image-based position that is determined to be invalid.

In step S3600, as the positioning sensor-based position of the second coordinate system and the positioning sensor-based speed of the input layer pass through the artificial neural network, the coordinates of the image-based position of the first coordinate system may be converted.

The athlete tracking server 1000 may determine the location of the sports participant based on the value output in step S3600.

According to the player tracking method according to an embodiment disclosed in the present application described above, by first tracking the player based on image data, the action or situation of the sports participant can be easily recognized and the position measurement with relatively high accuracy The advantage of image data that can implement can be utilized. However, in image data, there are situations in which it is difficult to distinguish objects due to occlusion events and the like, and data processing efficiency may be somewhat reduced because the data size is relatively large. These disadvantages of image data can be supplemented by positioning sensor data. .

Determining the position of sports participants from the positioning sensor data obtained from the positioning sensor device makes it easy to distinguish among sports participants, easily calculates the positions of sports participants even at night, and the data capacity is relatively small. Therefore, there is an advantage in that the amount of computation is small. In addition, there is convenience in that it is not necessary to install the image capture devices in advance on the periphery of the playfield. However, sports participants need to wear a positioning sensor device, may be greatly affected by structures around the playfield, and it may be relatively complicated to identify sports participants' actions or event recognition compared to image data.

On the other hand, tracking sports participants' motions from image data obtained from an image capture device has the advantage of not being affected by structures around the playfield and being easy to recognize sports participants' actions or events.

Therefore, according to another example of the player tracking method according to an embodiment of the present application, the position of a sports participant is determined from the positioning sensor data, but only when the position of the sports participant calculated from the positioning sensor data is not valid, image data The position of the player may be determined by further considering the position of the sports participant calculated from .

A player tracking method according to another example of an embodiment of the present application may be implemented by the player tracking system 100 shown in FIG. 3 . Also, the player tracking method according to an embodiment of the present application may be implemented by the player tracking server 1000 shown in FIG. 4 .

See Figure 20. 20 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application. Referring to FIG. 20 , the player tracking method according to an embodiment of the present application includes acquiring image data (S4100), acquiring positioning sensor data (S4200), and determining a player's position (S4300). can include

In steps S4100, S4200, and S4300, the contents of steps S1100, S1200, and S1300 described with reference to FIG. 16 may be equally applied. Accordingly, hereinafter, descriptions will be made focusing on contents of FIG. 21 that are different from or added to those of FIGS. 16 to 19 .

See Figure 21. FIG. 21 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application, and is a detailed diagram showing the step S4300 of FIG. 20 in detail.

Referring to FIG. 21 , the step S4300 according to an embodiment of the present application includes determining whether a location based on a positioning sensor is valid (S4310) and verifying and predicting the validity of an image-based location (S4330 and S4340). , determining the player's position based on at least one of a positioning sensor-based position or an image-based position (S4320 or S4350) and outputting the player's position to the determined position (S4360).

In step S4310, it may be determined whether the positioning sensor-based position obtained from the positioning sensor data is valid.

As an example, the above-described player tracking server 1000 may evaluate the effectiveness of the positioning sensor-based position by considering the reliability data of the positioning sensor-based position included in the positioning sensor data. In detail, the positioning sensor data may include information on reliability (eg, DoP, SNR) related to a position based on the positioning sensor. In this case, the player tracking server 1000 may evaluate the effectiveness of the positioning sensor-based position by considering the reliability information obtained from the positioning sensor data and a predetermined threshold value related to the reliability. For example, when information related to reliability included in the positioning sensor data is greater than the predetermined threshold value, the player tracking server 1000 determines that the positioning sensor-based position of the player calculated from the positioning sensor data is valid. can On the other hand, if information related to reliability included in the positioning sensor data is smaller than the preset threshold, the player tracking server 1000 may determine that the positioning sensor-based position of the player calculated from the positioning sensor data is not valid. there is.

As another example, the player tracking server 1000 may evaluate the effectiveness of positioning sensor-based positions in consideration of internal disparity of positioning sensor-based positions acquired from positioning sensor data. Specifically, when the "change" between the positioning sensor-based position corresponding to the first viewpoint and the positioning sensor-based position corresponding to the second viewpoint changes more than a predetermined threshold value, the player tracking server 1000 determines the second It may be determined that the positioning sensor-based position calculated from the positioning sensor data corresponding to the viewpoint is invalid.

As another example, the player tracking server 1000 may evaluate the effectiveness of the positioning sensor-based position by considering the difference or separation between the positioning sensor-based position obtained from the positioning sensor data and the image-based position obtained from the image data. Details related to this will be described later in detail with respect to FIG. 22 .

In step S4320, if it is determined that the location based on the positioning sensor is valid from the player tracking server 1000 in step S4310, the position of the sports participant may be determined based on the location based on the positioning sensor.

On the other hand, if it is determined that the positioning sensor-based position is not valid from the player tracking server 1000 in step S4310, steps of verifying the validity of the image-based position (S4330 and S4340) may be additionally included.

The verifying the validity of the image-based location may include verifying the validity of the image-based location (S4330) and determining whether the image-based location is predicted to be valid (S4340).

In step S4330, at least one of whether an occlusion event has occurred from the image data, the severity of the occlusion event, the internal disparity of the image-based position obtained from the image data, and the vertical movement of the sports participant is determined. One can consider validating the image-based location obtained from the image data.

According to an embodiment, the player tracking server 1000 may detect an occlusion event or determine whether an occlusion event has occurred using various methods as described above, and based on this, an image-based location can be validated or predicted.

For example, an occlusion event may be detected or whether an occlusion event has occurred may be determined in consideration of an overlapping degree between bounding boxes corresponding to sports participants obtained from image data.

As another example, an occlusion event may be detected or whether an occlusion event has occurred may be determined in consideration of pixel data related to sports participants obtained from image data.

As another example, an occlusion event may be detected or whether an occlusion event has occurred may be determined in consideration of positions of sports participants obtained from positioning sensor data. In detail, it may be determined that an occlusion event has occurred when more than a certain number of sports participants are located within a predetermined area from a location based on a positioning sensor corresponding to a specific sports participant.

As another example, considering the positions of sports participants obtained from the line of sight and pose data of the image capture device and positioning sensor data, an occlusion event is detected or whether an occlusion event has occurred. can judge

In the player tracking method according to an embodiment of the present application, if an occlusion event is detected or it is determined that an occlusion event has occurred, it may be predicted that the image-based position is invalid in step S4340.

However, even if it is determined that an occlusion event is detected or generated, if the occlusion severity is not serious, it can be predicted that the image-based location is effective in step S4340. To this end, when an occlusion event occurs, the validity of the image-based location may be verified or predicted by additionally considering the severity of the occlusion event.

According to an embodiment, the player tracking server 1000 may determine the severity of the occlusion event using various methods as described above, and based on this, the validity of the image-based location may be verified or predicted. .

In the player tracking method according to an embodiment of the present application, if a serious occlusion event is detected or the detected occlusion is determined to be serious, it may be predicted that the image-based location is not valid in step S4340. On the other hand, if a mild occlusion event is detected or the detected occlusion is determined to be mild, it can be predicted that the image-based location is valid in step S4340.

According to an embodiment, the player tracking server 1000 may verify or predict the validity of the image-based position by considering the image-based position acquired from the image data and the vertical movement of the sports participant at the same time point.

Referring back to FIG. 9(a) , when there is substantially no vertical movement of the sports participant 1 included in the image data, the position (L1_image) of the sports participant 1 calculated from the image data A difference between L0 and the actual position L0 of the sports participant 110 may be relatively small. Accordingly, the player tracking server 1000 may predict or determine the position L1_image of the sports participant 1 obtained from the image data as valid.

On the other hand, referring to FIG. 9(b) again, when the vertical movement of the sports participant 1 included in the image data actually exists, the position of the sports participant 110 calculated from the image data. A difference between (L2_image) and the actual position L0 of the sports participant 1 may be relatively large. In other words, when the vertical movement of the sports participant 1 included in the image data actually exists, the position L2_image of the sports participant 1 calculated from the image data is at the actual position L0. There is a high probability that significant errors will occur. Accordingly, the player tracking server 1000 may predict or determine that the location (L2_image) of the sports participant 1 obtained from the image data is invalid.

In the player tracking method according to an embodiment of the present application, when a vertical motion of a sports participant is detected, it may be predicted in step S4340 that the image-based position obtained with respect to the temporal point in time at which the vertical motion is detected is invalid. . On the other hand, if the vertical movement of the sports participant is not detected, it can be predicted that the image-based position is effective in step S4340.

According to an embodiment, the player tracking server 1000 may verify or predict the validity of the image-based position by considering the internal disparity of the image-based position obtained from the image data.

Referring back to FIG. 10 , when the difference between the sports participant's location (Lt1_image) calculated from the first time point and the sports participant's location (Lt2_image) calculated from the second time point is greater than a predetermined threshold value, the player tracking server (1000) may determine that the sports participant's position (Lt2_image) calculated from the second viewpoint is invalid. On the other hand, when the difference between the position of the sports participant (Lt3_image) calculated from the third viewpoint and the position (Lt4_image) of the sports participant calculated from the fourth viewpoint is smaller than a predetermined threshold value, the player tracking server 1000 The sports participant's location (Lt4_image) calculated from 4 viewpoints may be determined to be valid.

In the player tracking method according to an embodiment of the present application, when the internal disparity of the image-based position is measured to be greater than a predetermined threshold value, it may be predicted that the image-based position is invalid in step S4340. On the other hand, when the internal disparity of the image-based location is measured to be smaller than a predetermined threshold value, it can be predicted that the image-based location is effective in step S4340.

According to the above description, if it is predicted that the image-based location is valid in step S4340, the player tracking server 1000 may determine the location of the player based on the image-based location (S4350).

On the other hand, if it is predicted that the image-based position is not valid in step S4340, the player tracking server 1000 may determine the position of the player based on the positioning sensor-based position rather than the image-based position (S4320).

In FIG. 21, if it is predicted that the image-based position is not valid in step S4340, the player tracking server 1000 may determine the position of the player based on the positioning sensor-based position rather than the image-based position (S4320). Not limited. Therefore, even when it is predicted that the image-based location is not valid in step S4340, the player tracking server 1000 determines the player's location by considering both the image-based location and the positioning sensor-based location (eg, weighting according to reliability). It is apparent that any suitable method may be applied, such as a method for determining.

In step S4360, the location of the sports participant may be output or updated by outputting the determined location of the sports participant. The location of the sports participant to be outputted may be a coordinate value corresponding to any suitable coordinate system.

According to the player tracking method according to an embodiment disclosed in the present application described above, by first tracking the player based on the positioning sensor data, data processing efficiency is increased and sports participants can be easily distinguished. Positioning sensor You can take advantage of data. However, the positioning sensor data has an advantage in that the positioning sensor data can compensate for the disadvantages of the positioning sensor data, since the accuracy of position calculation varies depending on the surrounding structure and it may be somewhat insufficient to recognize the sports participant's behavior or situation.

Determining the position of sports participants from the positioning sensor data obtained from the positioning sensor device makes it easy to distinguish among sports participants, easily calculates the positions of sports participants even at night, and the data capacity is relatively small. Therefore, there is an advantage in that the amount of computation is small. In addition, there is convenience in that it is not necessary to install the image capture devices in advance on the periphery of the playfield.

Determining the location of a sports participant from image data obtained from an image capture device can accurately measure the location of a sports participant, is relatively less affected by structures around the playfield, and recognizes or analyzes the motion of a sports participant. It is relatively easy to do, and has the advantage of being convenient to use indoors. In addition, since the sports participant does not have to wear the positioning sensor device, it may be convenient from the side of the sports participant.

Therefore, according to the player tracking method according to an embodiment of the present application, the location of a sports participant is determined by considering both the positioning sensor data and the image data, thereby enjoying advantages of both the positioning sensor data and the image data. In addition, by determining the position of the sports participant by generating and comparing a reliability index for each calculated positioning sensor-based location and image-based location, the sports participant may be more accurately tracked.

A player tracking method according to an embodiment of the present application may be implemented by the player tracking system 100 shown in FIG. 3 . Also, the player tracking method according to an embodiment of the present application may be implemented by the player tracking server 1000 shown in FIG. 4 .

The player tracking server 1000 according to an embodiment of the present application may generate a reliability index related to an image-based location and a reliability index related to a positioning sensor-based location. The player tracking server 1000 may be provided to generate a confidence index associated with an image based position and a confidence index associated with a positioning sensor based position.

In detail, the player tracking server 1000 may generate a reliability index for reliability related to a location based positioning sensor obtained from positioning sensor data. In detail, the player tracking server 1000 may generate a reliability index for a positioning sensor-based position based on a result of evaluating the validity of the positioning sensor-based position.

For example, the positioning sensor data may include information (eg, DoP, SNR) related to the reliability of the sensor-signal signal. At this time, the player tracking server 1000 may evaluate the effectiveness of the positioning sensor-based position based on information related to the reliability of the sensor-signal signal (eg, DoP, SNR), and based on this, the player tracking server 1000 may perform the A reliability index for a location based on a positioning sensor may be generated in consideration of reliability information (eg, DoP, SNR) of a sensor-signal signal.

As another example, the player tracking server 1000 may evaluate the effectiveness of the positioning sensor-based positions based on internal disparity of positioning sensor-based positions calculated from positioning sensor data. In this case, the player tracking server 1000 may generate a reliability index for the positioning sensor-based position based on a result of evaluating the validity of the positioning sensor-based position based on the internal disparity of the positioning sensor-based position.

The player tracking server 1000 may also generate a confidence index for the confidence associated with the image-based position obtained from the image data. Specifically, the player tracking server 1000 may generate a reliability index for the image-based location based on a result of evaluating the validity of the image-based location.

For example, the player tracking server 1000 may evaluate the validity of the image-based position obtained from image data according to whether an occlusion event is detected. In this case, the player tracking server 1000 may generate a reliability index for the image-based position based on a result of evaluating the validity of the image-based position according to whether an occlusion event is detected.

As another example, the player tracking server 1000 may evaluate the effectiveness of the image-based position obtained from image data in consideration of the vertical movement of the sports participant. In this case, the player tracking server 1000 may generate a reliability index for the image-based position based on a result of evaluating the effectiveness of the image-based position considering the vertical movement of the sports participant.

As another example, the player tracking server 1000 may evaluate the effectiveness of the image-based position based on the internal disparity of the image-based position calculated from image data. In this case, the player tracking server 1000 may generate a reliability index for the image-based location based on a result of evaluating the validity of the image-based location based on the internal disparity of the image-based location.

In this case, the player tracking server 1000 may determine the location of the sports participant by comparing the reliability index for the location based on the positioning sensor and the reliability index for the image-based location.

Here, in order to compare a reliability index for a positioning sensor-based location and a reliability index for an image-based location, standardization between reliability indices is required. Accordingly, the player tracking server 1000 according to an embodiment of the present application may standardize a reliability index related to an image-based location and a reliability index related to a positioning sensor-based location. In detail, the player tracking server 1000 may standardize a reliability index related to an image-based location and a reliability index related to a positioning sensor-based location.

Reliability index standardization by the player tracking server 1000 is performed by using any appropriate statistical method or standardization model, so that the reliability index for positioning sensor-based positions and the confidence index for image-based positions can be compared. Indexes can be calibrated or standardized.

See Figure 22. 22 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application.

Referring to FIG. 22 , the method for tracking a player according to an embodiment of the present application includes obtaining positioning sensor data and positioning sensor-based position (S5100), acquiring image data and image-based position (S5200), positioning sensor It may include determining the error possibility of at least one of the base position and the image-based position (S5300) and determining the position of the player (S5400, S5500 and S5600). At this time, the step of determining the position of the player is a step of determining the position of the player (S5600) by comparing the reliability index of the positioning sensor-based position and the image-based position (S5500) according to the error possibility determination result of the step S5300. ) and determining the position of the player based on one of a positioning sensor-based position and an image-based position (S5400).

In step S5100, positioning sensor data transmitted from the positioning sensor devices 210 and 220 may be obtained. Also, a positioning sensor-based position may be obtained from the positioning sensor data. The positioning sensor-based position may be obtained from position-related data included in positioning sensor data, or may be obtained from speed or acceleration-related data included in positioning sensor data.

In step S5200, image data transmitted from the image capture device 300 may be acquired. Also, an image-based location may be obtained from the image data. The image-based location may be obtained from a coordinate value of a pixel corresponding to a location of a sports participant included in image data.

In step S5300, it may be determined whether each of the positioning sensor-based position obtained in step S5100 and the image-based position acquired in step S5200 is valid. Accordingly, embodiments of the player tracking server 1000 that determines the validity of the image-based location and positioning sensor-based location described with reference to FIGS. 5 to 12 may be implemented in step S5400.

According to an embodiment, in the step S5300, an error possibility of at least one of the positioning sensor-based position and the image-based position may be determined.

For example, an error possibility of at least one of the positioning sensor-based position and the image-based position may be determined in consideration of a difference between the positioning sensor-based position and the image-based position.

For example, the player tracking server 1000 may calculate a disparity index between an image-based position of a sports participant obtained from image data and a positioning sensor-based position of a sports participant obtained from positioning sensor data according to a time variable. there is. In this case, the player tracking server 1000 may determine validity of at least one of the image-based position and the positioning sensor-based position based on the distance index.

In detail, the player tracking server 1000 may calculate a first distance index corresponding to a difference between the first image-based position obtained at the first point of view and the first positioning sensor-based position. In addition, the player tracking server 1000 may calculate a second distance index corresponding to a difference between a position based on a second image obtained at a second point in time and a position based on a second positioning sensor.

At this time, the player tracking server 1000 considers the "size" of the first distance index and the second distance index, and determines the positioning sensor-based position (eg, the first positioning sensor-based position, the second positioning sensor-based position) and the image. The effectiveness of at least one of the base locations (eg, the first image-based location and the second image-based location) may be evaluated. For example, when the first distance index exceeds a first predetermined threshold value, the player tracking server 1000 determines that at least one of a first image-based position and a first positioning sensor-based position obtained at a first point in time is It can be judged that an error exists. On the other hand, if the second separation index is less than the predetermined second threshold, the player tracking server 1000 may determine that at least one of the second image-based position and the second positioning sensor-based position obtained at the second point in time is valid. can In this case, the first threshold value and the second threshold value may be the same value, but may be predetermined as different values.

In addition, the player tracking server 1000 determines the positioning sensor-based position (eg, the first positioning sensor-based position, the second positioning sensor-based position) and the image in consideration of the "change" of the second distance index for the first distance index. The effectiveness of at least one of the base locations (eg, the first image-based location and the second image-based location) may be evaluated.

For example, when the change values of the first separation index and the second separation index exceed a predetermined third threshold value (preferably, the first image-based position related to the first separation index and the first positioning sensor-based position are valid ), the player tracking server 1000 may determine that an error exists in at least one of the location based on the first image obtained at the second viewpoint and the location based on the first positioning sensor. On the other hand, when the change value of the first separation index and the second separation index is less than a predetermined fourth threshold value (preferably, the first image-based position related to the first separation index and the first positioning sensor-based position are determined to be valid). case), the player tracking server 1000 may determine that at least one of the location based on the second image and the location based on the second positioning sensor obtained at the second viewpoint is valid. In this case, the third threshold and the fourth threshold may have the same value, but may be predetermined as different values.

Determining the error possibility of at least one of the image-based position and the positioning sensor-based position in consideration of the above-described difference between the image-based position and the positioning sensor-based position is only an example, and the difference between the image-based position and the positioning sensor-based position The validity of the image-based location or positioning sensor-based location may be determined using any suitable method contemplated.

If it is determined in step S5300 that there is no possibility of error in at least one of the positioning sensor-based position and the image-based position, that is, if it is determined that both the positioning sensor-based position and the image-based position are valid, in step S5400, the player tracking server ( 1000) may determine the location of a sports participant based on one of a positioning sensor-based location and an image-based location.

Alternatively, since the positioning sensor-based position and the image-based position have a low possibility of error, the player tracking server 1000 may determine the position of the sports participant by considering both the positioning sensor-based position and the image-based position. For example, the location of a sports participant may be determined as an average of a positioning sensor-based location and an image-based location. Alternatively, the location of the sports participant may be determined by considering the reliability index for each of the positioning sensor-based location and the image-based location.

If it is determined in step S5300 that there is an error possibility for at least one of the positioning sensor-based position and the image-based position, that is, when it is determined that at least one of the positioning sensor-based position and the image-based position is invalid, in step S5500, The step of comparing the first reliability index of the positioning sensor-based location and the second reliability index of the image-based location may be performed by the player tracking server 1000 .

In detail, the player tracking server 1000 may generate a first reliability index for a location based on a positioning sensor in consideration of reliability information included in positioning sensor data. Alternatively, the player tracking server 1000 may generate a first reliability index for the positioning sensor-based position based on a result of determining the validity of the positioning sensor-based position based on the internal disparity of the positioning sensor-based position.

In addition, the player tracking server 1000 determines the validity of the image-based location based on at least one of an occlusion event related to the image-based location, vertical movement of the sports participant, and internal disparity, and determines the validity of the image-based location. A second confidence index for the location may be generated.

Also, the player tracking server 1000 may standardize the first reliability index and the second reliability index.

At this time, in steps S5500 and S5600, the location of the sports participant may be determined by the player tracking server 1000 comparing the first reliability index for a positioning sensor-based location and the second reliability index for an image-based location. there is.

For example, the player tracking server 1000 may determine the location of a sports participant based on a location corresponding to a high reliability index among the first reliability index and the second reliability index.

For example, when the first reliability index is greater than the second reliability index, that is, when the positioning sensor-based position corresponding to the first reliability index has relatively high reliability, the player tracking server 1000 determines the positioning sensor-based position Based on the position of the sports participant can be determined.

As another example, when the second reliability index is greater than the first reliability index, that is, when the image-based location corresponding to the second reliability index has relatively high reliability, the player tracking server 1000 determines the image-based location. Based on this, the position of the sports participant can be determined.

As another example, the player tracking server 1000 may determine the position of the sports participant by applying appropriate weights to the positioning sensor-based position and the image-based position in consideration of the first reliability index and the second reliability index.

For example, if the first reliability index is greater than the second reliability index, the player tracking server 1000 may determine the location of the sports participant by assigning a relatively high weight to the positioning sensor-based location.

For example, if the second reliability index is greater than the first reliability index, the player tracking server 1000 may determine the location of the sports participant by assigning a relatively high weight to the image-based location.

For example, if the first reliability index is similar to or substantially the same as the second reliability index, the player tracking server 1000 assigns similar weights to the positioning sensor-based location and the image-based location (eg, 1:1 weighting factor). average) to determine the position of the sports participant.

According to the player tracking method according to an embodiment disclosed in the present application described above, the position of a sports participant can be accurately measured by determining the possibility of an error in at least one of a positioning sensor-based position and an image-based position.

In addition, even when an error may exist in at least one of the positioning sensor-based position and the image-based position, by evaluating the reliability of the positioning sensor-based position and the image-based position, the position of the sports participant is accurately measured even when an error may exist. There are advantages to being able to do it.

According to the foregoing, in the step S5300, the content of determining validity considering the difference between the image-based position and the positioning sensor-based position has been mainly described, but this is only an example, and the image-based positioning and positioning sensor related to FIGS. 5 to 12 Contents for determining the validity of the base location may be equally applied.

In acquiring the location of the sports participant, accuracy of the location of the sports participant may vary according to the area of the playfield where the sports participant is located. In other words, depending on the area of the playfield where the sports participant is located, the reliability of the obtained position may change. This is because reliability associated with sensor-signals or image data may change for each area of the playfield where sports participants are located.

For example, the reliability of a sports participant's location obtained from positioning sensor data may be affected by the structure or shape of a stadium around a playfield where a sports participant is located. Therefore, since the degree of being influenced by the structure or shape of the stadium may be different for each playfield area, the reliability of the location of the sports participant obtained from positioning sensor data may be different for each area of the playfield.

In addition, reliability of the position of a sports participant obtained from image data may be affected by factors such as the position of an image capturing device, an angle, lens distortion, and the like. Accordingly, the reliability of the location of the sports participant obtained from the image capture device may be different for each playfield area.

Accordingly, another example of a player tracking method according to an embodiment disclosed in the present application may generate a reliability map for each playfield area and use reliability information according to a playfield area where a sports participant is located. Therefore, according to the method for tracking a player according to an embodiment disclosed in the present application, the position of a sports participant with improved accuracy can be obtained.

A player tracking method according to an embodiment of the present application may be implemented by the player tracking system 100 shown in FIG. 3 . Also, the player tracking method according to an embodiment of the present application may be implemented by the player tracking server 1000 shown in FIG. 4 .

The player tracking server 1000 according to an embodiment of the present application may generate a reliability index for each region of the playfield associated with an image-based position and a reliability index for each region of the playfield associated with a location based on a positioning sensor. In addition, the player tracking server 1000 may generate reliability maps based on the reliability index for each region of the playfield associated with the image-based position and the reliability index for each region of the playfield associated with the positioning sensor-based position. Specifically, the player tracking server 1000 may be provided to generate a reliability map based on a reliability index for each area of the playfield associated with the generated image-based position and a reliability index for each area of the playfield associated with the positioning sensor-based position. there is.

The player tracking server 1000 may generate a reliability index related to a location of a sports participant obtained from image data for each area of the playfield. In this case, the player tracking server 1000 may generate a reliability index for the image-based position for each area of the playfield by considering the result of evaluating the validity of the image-based position and the position of the sports participant in the playfield.

Regarding determining the validity of the image-based location, the contents described with respect to FIGS. 5 to 10 may be similarly applied.

The player tracking server 1000 may generate a reliability index related to a location of a sports participant obtained from positioning sensor data for each area of the playfield. In this case, the player tracking server 1000 may generate a reliability index for the location based on the positioning sensor for each area of the playfield by considering the result of evaluating the effectiveness of the position based on the positioning sensor and the position of the sports participant in the playfield.

Regarding determining the validity of the positioning sensor-based position, the contents described with reference to FIGS. 11 and 12 may be similarly applied.

In addition, the player tracking server 1000 may be provided to perform standardization of a reliability index related to an image-based location and a reliability index related to a positioning sensor-based location. The standardization of the confidence index by the player tracking server 1000 may use any suitable statistical method or standardization model to compare the confidence index for positioning sensor-based positions with the confidence index for image-based positions for each confidence index. can be calibrated or standardized.

In addition, the player tracking server 1000 may generate a first reliability map based on the reliability index for each playfield area associated with the generated and standardized positioning sensor-based location. In this case, the first reliability map may include information on a reliability index related to a location based on a positioning sensor obtained according to an area of a playfield.

See Figure 23. 23 is an exemplary diagram of a first reliability map according to an embodiment of the present application. Referring to FIG. 23 , the first reliability map may include reliability information related to a location based on a positioning sensor.

A plurality of areas including a first area R1_sensor, a second area R2_sensor, a third area R3_sensor, and a fourth area R4_sensor included in the first reliability map are in the plurality of areas of the playfield. It may be a corresponding area.

In addition, reliability information related to a location based on a positioning sensor may be different for each of a plurality of regions included in the first reliability map.

For example, reliability information related to the first region R1_sensor of the first reliability map may have a relatively high value. For example, considering the structure of a stadium around the playfield, when a sports participant is located in the center of the playfield corresponding to the first region R1_sensor, a highly reliable sensor-signal may be received.

As another example, reliability information related to the second region R2_sensor of the first reliability map may have a relatively low value. For example, considering the structure of the stadium around the playfield, when a sports participant is located at the edge of the playfield corresponding to the second area R2_sensor, the sensor-signal is affected by the structure of the stadium. Therefore, a sensor-signal with relatively low reliability can be received.

As another example, the reliability information related to the third area (R3_sensor) or the fourth area (R4_sensor) of the first reliability map is lower than the reliability information related to the first area (R1_sensor), and the reliability information related to the second area (R3_sensor) It may be a higher value than reliability.

However, the above-described reliability for each region of the first reliability map is merely an example for convenience of description, and it is obvious that the reliability for each region may change depending on the structure and shape of the stadium.

In addition, the player tracking server 1000 may generate a second reliability map based on the reliability index for each region of the playfield associated with the generated and standardized image-based location.

In this case, the second reliability map may include information on a reliability index related to an image-based location obtained according to a region of a playfield.

See Figure 24. 24 is an exemplary diagram of a second reliability map according to an embodiment of the present application. Referring to FIG. 24 , the second reliability map may include reliability information related to an image-based location.

A plurality of regions including the first region R1_image, the second region R2_image, the third region R3_image, the fourth region R4_image, and the fifth region R5_image included in the second reliability map play It may be an area corresponding to a plurality of areas of the field.

Also, reliability information related to an image-based location may be different for each of a plurality of areas included in the second reliability map.

For example, reliability information related to the first region R1_image of the second reliability map may have a relatively high value. For example, considering a pose including the position and angle of the image capture device 300, when a sports participant is located in an area of a playfield corresponding to the first area R1_image, an image with relatively little distortion. data can be obtained. Accordingly, when a sports participant is located in an area of the playfield corresponding to the first area R1_image, the reliability of the image-based location obtained from image data may be relatively high.

As another example, reliability information related to the second region R2_image of the second reliability map may have a relatively low value. For example, considering a pose including the position and angle of the image capture device 300, when a sports participant is located in an area of a playfield corresponding to the second area R2_image, an image with relatively much distortion. data can be obtained. Accordingly, when a sports participant is located in the area of the playfield corresponding to the second area R2_image, the reliability of the image-based location obtained from image data may be relatively low.

As another example, the reliability information related to the third area (R3_image), fourth area (R4_image) or fifth area (R5_image) of the second reliability map may include the position and angle of the image capture device 300 in a pose. Considering , it may be lower than the reliability associated with the first region R1_image of the second reliability map and higher than the reliability associated with the second region R2_image.

However, the above-described reliability for each region of the second reliability map is merely an example for convenience of description, and the reliability for each region may change depending on the position and pose of the image capture device 300 and the arrangement, number, distortion of the lens, and the like. It is self-evident that there is

See Figure 25. 25 is a flowchart illustrating a method for tracking a player according to an embodiment of the present application.

Referring to FIG. 25 , in the player tracking method according to an embodiment of the present application, positioning sensor data and a positioning sensor-based location are acquired (S6100), and positioning sensor-based location reliability for each region is obtained (S6200). , Acquiring image data and image-based location (S6300), obtaining reliability for each area of the image-based location (S6400), generating a reliability map (S6500), and based on the reliability of the reliability map area, the player's A step of determining a location (S6600) may be included.

In the step S6100, positioning sensor data transmitted from the positioning sensor devices 210 and 220 may be obtained. Also, a positioning sensor-based position may be obtained from the positioning sensor data. The positioning sensor-based position may be obtained from position-related data included in positioning sensor data, or may be obtained from speed or acceleration-related data included in positioning sensor data.

In the step S6200, reliability for each region of the positioning sensor-based location may be obtained.

In this case, the reliability for each region of the location based on the positioning sensor may be generated by the player tracking server 1000 . For example, as described above, the player tracking server 1000 may generate a reliability index related to a location of a sports participant obtained from positioning sensor data for each area of a playfield. In this case, the player tracking server 1000 may generate a reliability index for the location based on the positioning sensor for each area of the playfield by considering the result of evaluating the effectiveness of the position based on the positioning sensor and the position of the sports participant in the playfield.

In addition, the reliability for each area of the location based on the positioning sensor may be a reliability index standardized by the player tracking server 1000 .

The player tracking server 1000 may generate a reliability map related to the positioning sensor-based position based on the acquired reliability for each area of the positioning sensor-based position.

In step S6300, image data transmitted from the image capture device 300 may be obtained. Also, an image-based location may be obtained from the image data. The image-based location may be obtained from a coordinate value of a pixel corresponding to a location of a sports participant included in image data.

In step S6400, reliability for each region of the image-based location may be obtained.

In this case, reliability for each region of the image-based location may be generated by the player tracking server 1000 . For example, as described above, the player tracking server 1000 may generate a reliability index associated with a location of a sports participant obtained from image data for each area of a playfield. In this case, the player tracking server 1000 may generate a reliability index for the image-based position for each area of the playfield by considering the result of evaluating the validity of the image-based position and the position of the sports participant in the playfield.

Also, the reliability for each region of the image-based location may be a standardized reliability index.

The player tracking server 1000 may generate a reliability map related to the image-based location based on the acquired reliability for each region of the image-based location.

In the step S6500, the player tracking server 1000 considers the regional reliability of the positioning sensor-based position and the regional reliability of the image-based position, and sets a first reliability map related to the positioning sensor-based position and a second reliability map related to the image-based position. Confidence maps can be created.

As described above, it has been described that the player tracking server 1000 generates a first reliability map related to a positioning sensor-based location and a second reliability map related to an image-based location, respectively, but this is only an example, and the player tracking server ( 1000) may generate a single reliability map considering both the regional reliability of the positioning sensor-based location and the regional reliability of the image-based location.

In step S6600, the player tracking server 1000 considers the position of the sports participant in consideration of the reliability information included in the first reliability map related to the positioning sensor-based position and the second reliability map related to the image-based position generated in step S7500. can decide

To this end, the location of the sports participant corresponding to each of the regions of the first reliability map and the second reliability map should be additionally considered.

Accordingly, the player tracking server 1000 may be implemented to additionally acquire information about a playfield area where sports participants are located.

For example, the player tracking server 1000 may obtain data related to the position of the sports participant from the positioning sensor data, and the player tracking server 1000 may obtain data related to the position of the sports participant included in the positioning sensor data. , it is possible to detect a playfield area where a sports participant is located.

As another example, the player tracking server 1000 may obtain data related to the location of the sports participant from the image data, and the player tracking server 1000 may obtain data related to the location of the sports participant included in the image data. , it is possible to detect the playfield area where the sports participant is located.

Sequentially, the player tracking server 1000 determines the position of the sports participant by considering the first reliability information of the first reliability map area and the second reliability information of the second reliability map area corresponding to the detected area where the sports participant is located. can

For example, the player tracking server 1000 may determine the location of a sports participant in consideration of a location obtained from data having higher reliability information among the first reliability information and the second reliability information.

For example, when the sports participant is located in an area where the first reliability information is higher than the second reliability information, the player tracking server 1000 may determine the location of the sports participant based on the positioning sensor-based location.

For another example, when the sports participant is located in an area where the second reliability information is higher than the first reliability information, the player tracking server 1000 may determine the location of the sports participant based on the image-based location.

As another example, the player tracking server 1000 may determine the position of the sports participant by assigning weights according to the first reliability information and the second reliability information.

For example, when a sports participant is located in an area where the first reliability information is higher than the second reliability information, the player tracking server 1000 assigns a relatively higher weight to the positioning sensor-based position than to the image-based position, thereby determining sports The location of the participant can be determined.

As another example, when a sports participant is located in an area where the second reliability information is higher than the first reliability information, the player tracking server 1000 assigns a relatively higher weight to the image-based position than to the positioning sensor-based position, thereby The position of the sports participant can be determined.

The reliability map described above may be generated or updated in real time during a sports game. However, this is just an example, and a reliability map related to an image-based location and a positioning sensor-based location may be generated in advance even before a sports game starts.

In the foregoing, an embodiment of obtaining an image-based position from at least one image capture device and increasing the accuracy of a sports participant's position by considering the positioning sensor-based position has been described.

Hereinafter, an athlete tracking method, an athlete tracking device, and an athlete tracking system according to an embodiment that increases the accuracy of a sports participant's position based on a plurality of image data obtained from a plurality of image capture devices will be described in detail. In the player tracking system 100 according to the present embodiment, the description is focused on increasing the accuracy of the position of a sports participant based on a plurality of image data obtained from a plurality of image capture devices, but the above-described positioning sensor-based positions are also described together. It is self-evident to combine or combine the considerations.

26 is a schematic diagram of a player tracking device, a player tracking system, and a player tracking method according to an embodiment of the present application.

Referring to FIG. 26 , the player tracking method, player tracking device, and player tracking system according to an embodiment of the present application may track a player position using image data obtained from a plurality of image capture devices.

Hereinafter, the player tracking system 100 according to an embodiment of the present application will be described with reference to FIG. 26 .

Referring to FIG. 26 , the player tracking system 100 may include a plurality of image capture devices 301 , 302 , 303 , and 304 , a positioning sensor device 200 and a player tracking device 2000 . Meanwhile, although FIG. 26 does not show the intermediate server 400 or the playfield shown in FIG. 3, this is for convenience of description and image data or Positioning sensor data may be acquired.

The plurality of image capture devices 301, 302, 303, and 304 of the player tracking system 100 may photograph the playfield and the sports participant 1, and the player tracking device 2000 may include a plurality of image capture devices 301, 302, 303, 304), the position of the sports participant 1 may be determined based on the obtained image data, so that the location of the sports participant 1 may be continuously tracked.

Referring to FIG. 26 , a plurality of image capture devices 301, 302, 303, and 304 may be positioned around the playfield. The plurality of image capture devices 301, 302, 303, and 304 capture image data including the plurality of sports participants 1 and the playfield, and the player tracking server 2000 according to an embodiment of the present application can be sent to

A plurality of image capture devices 301, 302, 303, and 304 may be disposed at different locations around the playfield as shown in FIG. 26 . However, this is only an example, and the plurality of image capture devices 301, 302, 303, and 304 may be disposed at substantially the same location or adjacent locations around the playfield. In addition, it is obvious that the plurality of image capture devices 301, 302, 303, and 304 may be arranged in various arrangements at any suitable location.

At this time, the player tracking device 2000 may additionally acquire arrangement information related to positions and poses of the plurality of image capture devices and data related to lens distortion parameters of the plurality of image capture devices, from the plurality of image capture devices. there is. Using this, image data capable of better detecting the target sports participant or obtaining the location of the target sports participant more accurately can be selected. This will be described later in detail with respect to FIGS. 27 and 28 .

At this time, each image data obtained from the plurality of image capture devices may be labeled with information about the captured image capture device and transmitted to the player tracking device 2000 . For example, any suitable labeling technique may be applied such that the first image data obtained from the first image capture device 301 includes any suitable data indicating that the first image capture device 301 was taken. Further, any suitable labeling technique may be applied such that the second image data obtained from the second image capture device 302 includes any suitable data indicating that it was taken from the second image capture device 302 . Through this, the sports participant may be tracked by determining an image capture device capable of obtaining image data capable of better detecting the sports participant or acquiring the location of the sports participant more accurately through occlusion prediction. This will be described later in detail with reference to FIGS. 29 to 30 .

At this time, the plurality of sports participants 1 may wear the positioning sensor device 200, and the player tracking server 2000 may obtain positioning sensor data as described above with reference to FIG. 3 . Each positioning sensor data associated with a plurality of sports participants 1 may be considered for detecting whether or not occlusion is present, generating a virtual playfield, or predicting occlusion. This will be described later in detail with reference to FIGS. 27 to 31 .

The player tracking device 2000 may be implemented as any suitable type of server device. In the following description, the player tracking device 2000 is referred to as the player tracking server 2000, but this is merely for convenience of explanation, and the player tracking device 2000 does not necessarily have to be implemented in a server form.

Hereinafter, the player tracking server 2000 according to an embodiment of the present application will be described.

According to an embodiment of the present specification, the player tracking server 2000 may acquire a plurality of image data from a plurality of image capture devices 301, 302, 303, and 304 to determine the location of a sports participant. Also, the player tracking server 2000 may select an image including a target sports participant to be tracked as a candidate image from the obtained plurality of image data. Also, the player tracking server 2000 may select an effective image from a plurality of acquired candidate images in consideration of an occlusion event. Player tracking server 2000 may also locate a target sport participant from the selected valid image.

The content described with respect to FIG. 4 may be equally applied to the player tracking server 2000 according to an embodiment of the present specification. That is, the player tracking server 2000 may include a communication module 1100 , a memory 1200 and a controller 1300 . In particular, the contents of the communication module 1100, the memory 1200, and the controller 1300 may be equally applicable to the contents related to FIG. 4, and detailed descriptions thereof will be omitted. Therefore, hereinafter, the content added to the player tracking server 2000 will be mainly described.

Hereinafter, several operations performed by an embodiment of the player tracking system 100 in relation to the player tracking method will be described in more detail.

The player tracking system 100 according to this embodiment may acquire a plurality of image data from a plurality of image capture devices. Specifically, the player tracking server 200 may acquire a plurality of image data taken from a plurality of image capture devices.

The player tracking system 100 according to an embodiment of the present application may select a candidate image from a plurality of image data. In detail, the player tracking server 2000 may be implemented to detect a target sports participant to be tracked for each of a plurality of image data obtained from the plurality of image capture devices 301, 302, 303, and 304. In detail, each of the plurality of image data may include data (eg, pixels) corresponding to a plurality of sports participants. At this time, the player tracking server 2000 may be implemented to recognize sports participants in consideration of data corresponding to the plurality of sports participants described above, and data related to identifiers corresponding to the recognized sports participants. It is possible to detect a target sports participant by additionally considering.

Also, the player tracking server 2000 may be implemented to select at least one or more candidate images from the plurality of image data according to a detection result of the target sports participant. For example, the player tracking server 2000 may be implemented to select image data in which the target sports participant is detected among a plurality of image data as a candidate image. The candidate image may be at least one.

The player tracking system 100 according to one embodiment of the present application may be implemented to select a valid image that may be considered for determining the location of a target sports participant. Specifically, the player tracking server 2000 may be implemented to select an effective image that can be considered for determining the location of a target sports participant from among the selected at least one candidate image.

Here, a valid image may be used as a meaning encompassing images that may be considered for determining the location of a target sports participant. For example, all images in which target sports participants are detected may be valid images. However, according to a preferred embodiment, in order to more accurately determine the position of a target sports participant, the effective image is an image in which an occlusion event is not detected in relation to the target sports participant, or an occlusion event is detected but an occlusion event is not detected. The severity of can be defined as an image that is not serious.

For example, the player tracking server 2000 selects at least one valid image from the at least one candidate image in consideration of a result of determining whether an occlusion event has been detected for a plurality of image data or whether an occlusion event has occurred. You can choose.

For example, the player tracking server 2000 may be implemented to select candidate images in which no occlusion event related to a target sports participant is detected as valid images.

For another example, the player tracking server 2000 may be implemented to select, as valid images, candidate images in which an occlusion event related to a target sports participant is detected but the severity is not severe and mild.

As another example, the player tracking server 2000 determines whether the target sports participant has occluded other sports participants or whether the target sports participants have been occluded by other sports participants, with respect to the candidate image in which the occlusion event related to the target sports participant is detected. It can be implemented to select a valid image in consideration of whether or not it is lost.

Specifically, even if an occlusion event related to the target sports participant is detected in the first candidate image, if the occlusion event is detected because the target sports participant occludes another sports participant, the position of the target sports participant from the first candidate image Determining may be relatively accurate.

On the other hand, even if an occlusion event related to the target sports participant is detected in the second candidate image, if the occlusion event is detected because the target sports participant is occluded by another sports participant, the position of the target sports participant is detected from the second candidate image. can be relatively difficult to determine. Also, the accuracy of the location of the target sports participant obtained from the second candidate image may be relatively low.

Accordingly, the player tracking server 2000 considers whether an occlusion event is detected because a target sports participant occludes another sports participant or a target sports participant occludes another sports participant in the candidate image, and selects a valid image. can be implemented to select Preferably, the operation of the player tracking server 2000 may be advantageous when an occlusion event related to a target sports participant is detected in most or all candidate images.

As another example, the player tracking server 2000 may be implemented to select an effective image in consideration of arrangement information related to positions and poses of the plurality of image capture devices 301 , 302 , 303 , and 304 . Specifically, the plurality of image capture devices 301, 302, 303, and 304 may each be disposed in any suitable arrangement at different locations around the playfield. At this time, the player tracking server 2000, among the plurality of image data obtained from the plurality of image capture devices (301, 302, 303, 304), the arrangement of the plurality of image capture devices (301, 302, 303, 304) In consideration of the information, it may be implemented to select the image data with the largest or the most central image of the target sports participant as a valid image.

As another example, the player tracking server 2000 may be implemented to select an effective image in consideration of a lens distortion parameter. Specifically, the center of the image data has a high possibility of relatively less distortion when considering the lens distortion parameter. Accordingly, when the location of the target sports participant is calculated from image data in which the target sports participant is located in the center of the image data, the accuracy of the location may be relatively high. On the other hand, the edges of the image data are likely to have a relatively large amount of distortion when considering the lens distortion parameter. Accordingly, when the location of the target sports participant is calculated from the image data located at the edge of the image data, the accuracy of the location may be relatively low. Accordingly, the player tracking server 2000 may be implemented to select, as an effective image, image data having relatively little distortion of an area of the image data where the target sports participant is located, in consideration of the lens distortion parameter.

See Figure 27. 27 is a diagram illustrating an exemplary process of selecting a valid image of the player tracking server 2000 according to an embodiment of the present application.

27(a) may be a plurality of image data acquired from a plurality of image capture devices 301, 302, 303, and 304. In this case, the plurality of image data of FIG. 27(a) may be image data acquired at the same temporal point in time. The plurality of image data of FIG. 27(a) may include image data including the target sports participant whose position is to be determined and image data not including the target sports participant.

27(b) may be at least one candidate image in which a target sports participant is detected. In detail, the player tracking server 2000 may detect a target sports participant for each of the plurality of image data of FIG. 27 (a), and may select the image data from which the target sports participant is detected as a candidate image.

27(c) may be at least one valid image. Specifically, the at least one effective image may be an image in which no occlusion event related to a target sports participant is detected from among the candidate images of FIG. 27(b).

When there are a plurality of valid images as shown in FIG. 27(c), the player tracking server 2000 may determine the location of the target sports participant by considering the location of the target sports participant obtained from each of the valid images. For example, the player tracking server 2000 may determine the location of the target sports participant by averaging the location of the target sports participant obtained from each valid image.

For another example, when there are a plurality of valid images as shown in FIG. 27(c), the player tracking server 2000 additionally considers arrangement information of image capture devices or lens distortion parameters, etc. to determine the position of the target sports participant. can decide

27(d) is an example of at least a single image selected by the player tracking server 2000 in consideration of arrangement information of image capture devices or a lens distortion parameter.

The player tracking system 100 according to an embodiment of the present application may determine the location of a target sports participant based on the valid image. Specifically, the player tracking server 2000 according to an embodiment of the present application may determine the position based on pixel data (pixel position) corresponding to the target sports participant of the selected valid image.

As an embodiment, when there are a plurality of selected valid images, the player tracking server 2000 may determine the location of the target sports participant by considering all locations of the target sports participant obtained from the plurality of valid images.

Specifically, the effective image may include a first effective image and a second effective image. At this time, the player tracking server 2000 may obtain the first location of the target sports participant based on the pixel position corresponding to the target sports participant in the first valid image. Also, the player tracking server 2000 may obtain a second location of the target sports participant based on a pixel position corresponding to the target sports participant in the second valid image.

Finally, the player tracking server 2000 may determine the location of the target sports participant based on both the first location and the second location.

For example, the player tracking server 2000 may determine the location of the target sports participant by averaging the first location and the second location.

As another embodiment, when there are a plurality of selected valid images, the player tracking server 2000 may determine the location of the target sports participant in consideration of the locations of the target sports participant and the image capture device that captured the effective image. To this end, the player tracking server 2000 may be provided to select a single image from the available images, taking into consideration conditions related to the location of the target sports participant and the image capture device that captured the valid image. At this time, the player tracking server 2000 may determine the location of the target sports participant based on the single image.

For example, when the location of the target sports participant and the location of the image capture device capturing the effective image are relatively close, the location of the target sports participant acquired from the effective image is likely to be relatively accurate. On the other hand, when the position of the target sports participant and the image capture device that has taken the effective image are relatively far, the location of the target sports participant obtained from the effective image is likely to be relatively accurate.

Accordingly, when there are a plurality of selected valid images, the player tracking server 2000 may determine the location of the target sports participant by additionally considering conditions related to the location between the target sports participant and the image capture device that has captured the valid image.

As another embodiment, when there are a plurality of selected valid images, the player tracking server 2000 may determine the position of the target sports participant in consideration of a lens distortion parameter of an image capture device that has captured the valid images. To this end, the player tracking server 2000 may select a single image from valid images in consideration of conditions related to lens distortion parameters of the image capture device that has captured the valid images. At this time, the player tracking server 2000 may determine the location of the target sports participant based on the single image.

For example, in relation to the lens distortion parameter of an image capture device that has captured an effective image, when the target sports participant is located in an area in which the lens distortion in the effective image is relatively large (e.g., the edge of the effective image), The location obtained from the target sports participant may have low accuracy.

On the other hand, in relation to the lens distortion parameter of the image capture device that has captured the effective image, when the target sports participant is located in an area where the lens distortion in the effective image is relatively small (eg, the center of the effective image), the target sports participant The obtained location may have high accuracy.

Accordingly, when there are a plurality of selected valid images, the player tracking server 2000 may determine the position of the target sports participant by additionally considering conditions related to the lens distortion parameter of the image capture device that has captured the valid images.

The player tracking server 2000 according to an embodiment of the present application determines the location of a target sports participant in consideration of whether the target sports participant has covered other sports participants or whether the target sports participant has been covered by other sports participants. can Preferably, the location of the target sports participant is determined by taking into account whether the target sports participant is obscured by another sports participant or whether the target sports participant is obscured by another sports participant, when there is no valid image, that is, multiple It can be advantageously applied when a target sports participant is detected from the image data, but an occlusion event related to the target sports participant occurs in all candidate images. To this end, the player tracking server 2000 may select a target image from candidate images in consideration of whether the target sports participant is obscured by other sports participants or whether the target sports participant is obscured by other sports participants. In this case, the player tracking server 2000 may determine the position of the target sports participant based on the target image.

For example, if the target sports participant obscures another sports participant, a pixel location corresponding to the target sports participant in the image data may be specified. Accordingly, the player tracking server 2000 may select image data in which a target sports participant obscures another sports participant as a target image. Additionally, the player tracking server 2000 may determine the location of a target sports participant based on the selected target image.

On the other hand, if the target sports participant is obscured by another sports participant, there is a high possibility that a pixel position corresponding to the target sports participant in the image data is not specified. Accordingly, the player tracking server 2000 may be implemented so that image data in which a target sports participant is obscured by another sports participant is not used to determine the location of the target sports participant.

However, the above is only an example, and when an occlusion event related to a target sports participant occurs from all candidate images, the player's position is determined using the positioning sensor-based location described in FIGS. 16 to 25. It is self-evident that you can decide

In the above, the configuration and operation of the player tracking server 2000 according to an embodiment of the present application have been described. Hereinafter, the player tracking method according to the present embodiment will be described. In the following description, the player tracking method according to the present embodiment will be described as being performed by the player tracking system 100 described above. However, since this is only for convenience of description, the player tracking method according to the present embodiment is not limited to the player tracking system 100 described above. That is, the player tracking method to be described later does not necessarily have to be performed only by the player tracking system 100, and may be performed by other systems or devices having functions similar to those of the player tracking system 100 described above.

See Figure 28. 28 is a flowchart of a method for tracking a player according to an embodiment of the present application.

Referring to FIG. 28 , the player tracking method according to an embodiment of the present application includes acquiring a plurality of image data (S7100), selecting image data including a target sports participant (S7200), and selecting a valid image. (S7300), and determining the position of the target sports participant based on the valid image (S7400).

Hereinafter, each step described above will be described in more detail.

The player tracking server 2000 may acquire a plurality of image data. (S7100) Specifically, the player tracking server 2000 may acquire a plurality of image data from a plurality of image capture devices 301, 302, 303, and 304 through the communication module 1100.

In the step S7100, the player tracking server 2000 may obtain a plurality of image data captured from a plurality of image capture devices 301, 302, 303, and 304. At this time, the player tracking server 2000 provides arrangement information related to positions and poses of the plurality of image capture devices 301, 302, 303, and 304, identifiers of the image capture devices, lens distortion parameters, and positions of sports participants. Data related to (pixel data of image data or positioning sensor data) can also be obtained.

Each of the plurality of image data obtained in step S7100 may include a plurality of sports participants. Each of the plurality of image data may include a target sports participant whose position is to be determined, but may not include the target sports participant.

Accordingly, the player tracking server 2000 may detect a target sports participant from a plurality of image data and select image data including the target sports participant (S7200). Specifically, the player tracking server 2000 may be provided to perform an operation of detecting a target sports participant included in a plurality of image data. In this case, the player tracking server 2000 may be implemented to select a candidate image according to a detection result of a target sports participant.

For example, the player tracking server 2000 may select image data in which a target sports participant is detected as a candidate image.

In this case, at least one candidate image in which the target sports participant is detected may be present.

For example, a target sports participant may be detected from all of the plurality of image data. In this case, all images in which target sports participants are detected may be selected as candidate images.

However, the candidate images selected according to whether the target sports participant is detected may include images that are not suitable as a basis for determining the location of the target sports participant.

Accordingly, the player tracking server 2000 may select a valid image from candidate images. (S7300) Here, the effective image may be used as a meaning encompassing any suitable image for determining the location of the target sports participant.

Specifically, the player tracking server 2000 may be provided to select an effective image from the candidate images, considering whether or not an occlusion event related to a target sports participant is detected.

For example, the player tracking server 2000 may detect an occlusion event or determine whether an occlusion event has occurred. Alternatively, the player tracking server 2000 may determine the severity of the occlusion event.

In particular, the player tracking server 2000 may detect an occlusion event “related to a target sports participant” or determine whether an occlusion event has occurred. Alternatively, the player tracking server 2000 may determine the severity of the occlusion event "related to the target sports participant".

In this case, the player tracking server 2000 may select at least one valid image from the candidate images selected in step S7200 based on the detection result of the occlusion event related to the target sports participant.

For example, the player tracking server 2000 may select candidate images in which no occlusion event related to the target sports participant is detected as valid images.

On the other hand, the player tracking server 2000 may not select candidate images in which an occlusion event related to a target sports participant is detected as valid images. However, even when an occlusion event related to a target sports participant is detected, if it is determined that the occlusion event is not serious, the player tracking server 2000 may be provided to select a corresponding candidate image as a valid image. .

In the step S7300, a single image may be selected in consideration of additional conditions from the valid images selected in consideration of the detection result of the occlusion event.

Specifically, when there are a plurality of valid images selected in consideration of the detection result of an occlusion event related to the target sports participant, a condition related to the position between the target sports participant and the image capture device that captured the effective image or an image of the valid image. A single image may be selected from the valid images by additionally considering the conditions related to the lens distortion parameter of the capture device.

For example, the accuracy of the position of the target sports participant determined according to the size or sharpness of the target sports participant included in the effective image may be different. If the size of the target sports participant included in the valid image is large or the resolution is clear, the position of the target sports participant determined from the valid image will be relatively accurate.

In this case, the size or resolution of the target sports participant included in the effective image may depend on the location of the target sports participant and the location where the effective image is captured. For example, when the location of the target sports participant and the location where the effective image is captured are relatively close, the target sports participant may be captured large and clear. On the other hand, if the location of the target sports participant is relatively far from the location where the effective image is captured, the target sports participant may be relatively small and not clear.

Therefore, according to an embodiment, in step S7300, a single image may be selected from the valid images in consideration of conditions related to positions between the target sports participant and the image capture device that has captured the effective image.

As another example, the accuracy of the determined location of the target sports participant may be different according to the location of the target sports participant in the effective image and the degree of distortion for each region of the effective image.

For example, if the target sports participant is located in an area with little distortion of the effective image, the location of the target sports participant determined from the effective image will be relatively accurate.

For another example, when the target sports participant is located in the center of the effective image with a relatively small degree of distortion, the accuracy of the location of the target sports participant determined from the corresponding effective image may be relatively high. On the other hand, if the target sports participant is located at the edge where the degree of distortion is relatively high within the effective image, the accuracy of the position of the target sports participant determined from the corresponding effective image may be relatively low.

In this case, the degree of distortion for each region of the effective image may depend on the lens distortion parameter. Therefore, according to an embodiment, in step S7300, a single image may be selected as valid images in consideration of a condition related to a lens distortion parameter of an image capture device that has captured the valid image.

In step S7300, a target image may be selected from the candidate images selected in step S7200 in consideration of the type of occlusion event related to the target sports participant. Preferably, selecting the target image in consideration of the type of occlusion event related to the target sports participant may advantageously be applied to all candidate images when an occlusion event related to the target sports participant is detected. there is.

The types of the occlusion event may include a first type that occurs when a target sports participant occludes another sports participant and a second type that occurs when the target sports participant is occluded by another sports participant.

In the case of the first type, it may be relatively easy to specify a location of a pixel corresponding to a target sports participant. On the other hand, in the case of the second type, since the target sports participant is covered by other sports participants, it may be relatively difficult to specify the position of a pixel corresponding to the target sports participant.

Accordingly, the player tracking server 2000 may be provided to select a candidate image corresponding to the first type as a target image. On the other hand, the player tracking server 2000 may be provided not to select a candidate image corresponding to the second type as a target image.

The player tracking server 2000 may determine the location of the target sports participant based on the selected effective image. (S7400)

Specifically, the player tracking server 2000 determines the position of the target sports participant by considering the pixel position corresponding to the target sports participant in the valid image, target image, or single image (hereinafter, effective image) selected in step S7300. can be provided.

For example, if the number of valid images selected in step S7300 is singular, the location of the target sports participant may be determined by considering the pixel location of the selected image corresponding to the target sports participant.

As another example, if there are a plurality of valid images selected in step S7300, the position of the target sports participant may be determined by considering all (or some) pixel positions corresponding to the target sports participant in the plurality of valid images. For example, the location of a target sports participant may be determined by averaging pixel locations obtained from a plurality of valid images or the like.

Alternatively, when there are a plurality of valid images selected in the step S7300, the condition related to the position between the target participant and the camera that has taken the at least one valid image or the condition related to the lens distortion parameter described in the step S7300 With further consideration, the location of the target sports participant may be determined. For example, a pixel position corresponding to a target sports participant within a single image selected from an effective image, etc., in consideration of a condition related to a position between the target participant and a camera that has taken the at least one valid image, or a condition related to a lens distortion parameter. The position of the target sports participant may be determined in consideration of .

According to the foregoing, although the location of the target sports participant is determined based on locations obtained from image data, FIGS. 16 to 27 determine the location of the sports participant in consideration of the location obtained from positioning sensor data. It is obvious that the contents of can be applied.

Hereinafter, the player tracking server 2000 according to an embodiment of the present application will be described.

According to an embodiment of the present specification, the player tracking server 2000 may acquire image data from a plurality of image capture devices 301, 302, 303, and 304 to determine the location of a sports participant. Also, the player tracking server 2000 may obtain positioning sensor data from the positioning sensor device 200 to determine the position of a sports participant. Also, the player tracking server 2000 may calculate an image-based position from the acquired image data, and may calculate a positioning sensor-based position from the acquired positioning sensor data. Also, the player tracking server 2000 may create a virtual playfield or a matching table based on the calculated image-based position and positioning sensor-based position. However, according to a preferred embodiment, the player tracking server 2000 may create a virtual playfield or a matching table based on the positioning sensor-based location.

The content described with respect to FIG. 4 may be equally applied to the player tracking server 2000 according to an embodiment of the present specification. That is, the player tracking server 2000 may include a communication module 1100, a memory 1200, and a controller 1300. In particular, the contents of the communication module 1100, the memory 1200, and the controller 1300 may be equally applicable to the contents related to FIG. 4, and detailed descriptions thereof will be omitted. Therefore, hereinafter, description will be focused on the contents added to the player tracking server 2000.

Hereinafter, several operations performed by an embodiment of the player tracking system 100 in relation to the player tracking method will be described in more detail.

The player tracking system 100 according to this embodiment may acquire a plurality of image data from a plurality of image capture devices. Specifically, the player tracking server 200 may acquire a plurality of image data taken from a plurality of image capture devices. Also, the player tracking system 100 may obtain arrangement information related to positions and poses of a plurality of image capture devices from a plurality of image capture devices. Also, the player tracking system 100 may obtain positioning sensor data from the positioning sensor device 200 . In addition, the player tracking system 100 may generate a matching table by generating a virtual playfield based on arrangement information related to the image capture device and positioning sensor data.

The player tracking server 2000 according to an embodiment of the present application may determine the validity of the location of the sports participant obtained from the image data described above with reference to FIGS. 6 to 11 .

For example, the player tracking server 2000 detects an occlusion event for each of a plurality of image data acquired from the plurality of image capture devices 301, 302, 303, and 304 or determines whether an occlusion event has occurred. can judge

Also, the player tracking server 2000 may determine the severity of the occlusion event for each of a plurality of image data obtained from the plurality of image capture devices 301, 302, 303, and 304.

The player tracking system 100 according to this embodiment may create a virtual playfield. In detail, the player tracking server 2000 may generate a virtual playfield based on positioning sensor data and arrangement information related to an image capture device. Specifically, the player tracking server 2000 may generate a virtual point corresponding to the image capture device based on arrangement information related to the location of the image capture device.

Also, the player tracking server 2000 may generate a plurality of moving points respectively corresponding to the positions of a plurality of sports participants based on positioning sensor data of the plurality of sports participants.

The player tracking system 100 according to this embodiment may be implemented to predict an occlusion event (or severity of the occlusion event) based on the generated virtual playfield. Specifically, the player tracking server 2000 may be implemented to predict an occlusion event among sports participants based on positional relationships between a plurality of moving points in a virtual playfield and arrangement information related to a pose of an image capture device.

The player tracking system 100 according to this embodiment may be implemented to generate a matching table between a plurality of image capture devices and a plurality of sports participants based on the created virtual playfield.

Specifically, the player tracking server 2000 determines the plurality of image capture devices and the plurality of sports based on relative positions between a plurality of points corresponding to the image capture device of the created virtual playfield and a plurality of moving points corresponding to sports participants. Relations between participants can be calculated. In this case, the player tracking server 2000 may be implemented to generate a matching table between a plurality of image capture devices and sports participants based on the relationship and the prediction result of the occlusion event.

See Figure 29. 29 is an exemplary diagram illustrating a virtual playfield according to an embodiment of the present application.

The player tracking server 2000 may generate a virtual playfield based on arrangement information of image capture devices and location information of a plurality of sports participants.

The arrangement information of the image capture devices may include information about positions, lines of sight, and poses of the plurality of image capture devices 301 , 302 , 303 , and 304 .

Location information of the plurality of sports participants may be obtained from positioning sensor data. Alternatively, location information of the plurality of sports participants may be obtained from image data.

The player tracking server 2000 may generate a plurality of points corresponding to the plurality of image capture devices in a virtual playfield coordinate system in consideration of arrangement information of the image capture devices.

At this time, the positions of the image capture devices may change, but may be fixed.

When the positions of the image capture devices change, the player tracking server 2000 sets a plurality of moving points corresponding to the plurality of image capture devices in consideration of changes in location information of the plurality of image capture devices as the sports game progresses. It can be created in a virtual playfield coordinate system.

When the positions of the image capture devices are fixed, the player tracking server 2000, as shown in FIG. can be created in the playfield coordinate system of

In addition, the player tracking server 2000 may generate a plurality of points (MP1, MP2, MP3) corresponding to the plurality of sports participants in a virtual playfield coordinate system in consideration of location information of the plurality of sports participants. can

In addition, the player tracking server 2000 may generate a plurality of points corresponding to the plurality of sports participants in a virtual playfield coordinate system in consideration of changes in location information of the plurality of sports participants as the sports game progresses. . That is, the plurality of points corresponding to the plurality of sports participants may be moving points MP1, MP2, and MP3.

In addition, the player tracking server 2000 may calculate relationships between the plurality of image capture devices 301 , 302 , 303 , and 304 and the plurality of sports participants based on the generated virtual playfield.

Specifically, the player tracking server 2000 includes points FP1, FP2, FP3, and FP4 corresponding to the plurality of image capture devices 301, 302, 303, and 304 of the virtual playfield and points corresponding to a plurality of sports participants. Relations between the plurality of image capture devices 301, 302, 303, and 304 and the plurality of sports participants may be calculated based on the relative positions of the moving points MP1, MP2, and MP3.

In addition, the player tracking server 2000 determines whether or not an occlusion event occurs between a plurality of sports participants, based on the relationship between the plurality of image capture devices 301, 302, 303, and 304 and the plurality of sports participants. can predict

For example, from the points FP1, FP2, FP3, FP4 corresponding to the plurality of image capture devices 301, 302, 303, 304 of the virtual playfield, the plurality of moving points corresponding to the plurality of sports participants A plurality of virtual lines VL may be generated for (MP1, MP2, MP3).

At this time, the player tracking server 2000 determines the locations of the points FP1, FP2, FP3, and FP4 corresponding to the plurality of image capture devices 301, 302, 303, and 304, and the plurality of points corresponding to the plurality of sports participants. Whether or not an occlusion event occurs between sports participants may be predicted by considering at least one of positions of the moving points MP1 , MP2 , and MP3 and angles between the plurality of virtual lines VL1 and VL2 .

For example, referring to FIG. 29 , a first imaginary line VL1 extending from the first fixed point FP1 to the first moving point MP1 and a second moving point MP2 from the first fixed point FP1 ) may be generated.

At this time, when the first virtual line VL1 and the second virtual line VL2 overlap or the angle between the first virtual line VL1 and the second virtual line VL2 is less than a predetermined angle, The player tracking server 2000 determines or predicts that an occlusion event may occur between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2. can

At this time, the player tracking server 2000 additionally determines that an occlusion event is first fixed between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2. It may be determined or predicted that it may occur from the first image capture device 301 corresponding to the point FP1.

In addition, the player tracking server 2000 may predict whether an occlusion event occurs between sports participants by further considering the positional difference between the first moving point MP1 and the second moving point MP2.

For example, referring to FIG. 29 , even if the angles of the first virtual line VL1 and the second virtual line VL2 are the same, the first moving point MP1 and the second moving point MP2 According to the positional difference between the first fixed point FP1, the error between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2 Whether or not there is a clone may be different.

Accordingly, the player tracking server 2000 may predict whether or not an occlusion event occurs between sports participants by further considering the position difference between moving points corresponding to a plurality of sports participants.

In addition, the player tracking server 2000 includes a point of a virtual playfield corresponding to a plurality of image capture devices, a moving point of a virtual playfield corresponding to a plurality of sports participants, and a line of sight of a plurality of image capture devices. Occurrence of an occlusion event between sports participants may be predicted by considering information on sight.

For example, considering the line of sight of the first image capture device 301 corresponding to the first fixed point FP1, a virtual line corresponding to the line of sight may be generated. In this case, a corresponding line of sight may be generated. If the first moving point and the second moving point are substantially located on the imaginary line, the player location server 2000 determines the relationship between the first sports participant corresponding to the first moving point and the second sports participant corresponding to the second moving point. It may be determined that an occlusion event has occurred.

In addition, the player tracking server 2000 considers the velocity data (or acceleration data) of the plurality of sports participants and the pose information of the plurality of image capture devices, and determines whether an occlusion event occurs between the plurality of sports participants at a later point in time. Predictable.

For example, referring to FIG. 29 , velocity data (or acceleration data) of a first sports participant and a third sports participant may be obtained by any suitable method. Also, pose information of the first image capture device may be obtained. At this time, the player tracking server 2000 determines that the first moving point FP1 corresponding to the first sports participant is the third moving point FP3 corresponding to the third sports participant based on the velocity data (or acceleration data). It can be predicted that the third moving point FP3 corresponding to the third sports participant will move in a direction closer to the first moving point FP1 corresponding to the first sports participant.

Also, the player tracking server 2000 may predict a pose of the first fixation point FP1 corresponding to the first image capture device based on the pose information.

Accordingly, the player tracking server 2000 determines the first sports participant corresponding to the first moving point MP1 and the second moving point MP2 with respect to the first image capture device corresponding to the first fixed point FP1. ), occlusion at a later point in time between the second sports participants corresponding to ) may be predicted.

In addition, the player tracking server 2000 may include a plurality of image capture devices and a plurality of sports participants calculated based on relative positions between a plurality of points corresponding to a plurality of image capture devices and a plurality of moving points corresponding to a plurality of sports participants. It may be implemented to generate a matching table between a plurality of image capture devices and a plurality of sports participants based on the relationship between the plurality of image capture devices.

The player tracking server 2000 may generate a matching table between a plurality of image capture devices and a plurality of sports participants based on the occlusion prediction.

For example, a plurality of moving points (FP1, FP2, FP3, FP4) corresponding to a plurality of sports participants ( Virtual lines may be extended for each of a plurality of moving points including MP1, MP2, and MP3, and a table for a prediction result of an occlusion event based on an angle between the virtual lines may be generated.

For example, referring to FIG. 29 , between a first sports participant corresponding to the first moving point MP1 and a second sports participant corresponding to the second moving point MP2, a first fixed point FP1 is formed. An occlusion event from the first image capture device 301 corresponding to may be predicted.

On the other hand, between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2, the second, third, and fourth fixed points correspond to the second, third, and fourth fixed points. It can be predicted that no occlusion event will occur from the 4 image capture devices 302, 303, and 304.

At this time, the player tracking server 2000 corresponds to the positions of the plurality of fixation points FP1, FP2, FP3, and FP4 corresponding to the image capture devices 301, 302, 303, and 304 and the plurality of sports participants. A matching table may be generated based on whether or not occlusion is predicted according to the positions of the plurality of moving points and the angles between the plurality of virtual lines.

Similarly, the player tracking server 2000 may generate a matching table based on whether or not occlusion is predicted according to a positional difference between a plurality of moving points corresponding to a plurality of sports participants.

Similarly, the player tracking server 2000 may generate a matching table based on whether or not occlusion is predicted according to the velocity data (or acceleration data) of a plurality of sports participants and pose information of a plurality of image capture devices.

The matching table generated as described above can be used to continuously monitor by selecting an image capture device most suitable for tracking a target sports participant.

The matching table may also be utilized to select the most appropriate image capture device for photographing a target sports participant in connection with a broadcast.

The matching table can also be utilized to select the most suitable image capture device to create individual highlights of the target sports participant.

In the above, the configuration and operation of the player tracking server 2000 according to an embodiment of the present application have been described. Hereinafter, the player tracking method according to the present embodiment will be described. In the following description, the player tracking method according to the present embodiment will be described as being performed by the player tracking system 100 described above. However, since this is only for convenience of description, the player tracking method according to the present embodiment is not limited to the player tracking system 100 described above. That is, the player tracking method to be described later does not necessarily have to be performed only by the player tracking system 100, and may be performed by other systems or devices having functions similar to those of the player tracking system 100 described above.

See Figure 30. 30 is a flowchart illustrating a method for tracking a player using a plurality of image capture devices according to an embodiment of the present application.

Referring to FIG. 30 , the method for tracking a player using a plurality of image capture devices according to an embodiment of the present application includes obtaining arrangement information of a plurality of image capture devices (S8100), positioning sensor data of a plurality of sports participants. may include receiving (S8200), acquiring positions of a plurality of sports participants from positioning sensor data (S8300), generating a virtual playfield (S8400), and generating a matching table (S8500). there is.

Hereinafter, each step described above will be described in more detail.

The player tracking server 2000 may obtain arrangement information of a plurality of image capture devices. (S8100) Specifically, in step S8100, the player tracking server 2000 records a plurality of image data captured from a plurality of image capture devices 301, 302, 303, and 304 and a plurality of image capture devices 301 and 302. , 303, 304) may be acquired through the communication module 1100, arrangement information related to positions and poses.

The player tracking server 2000 may receive positioning sensor data of a plurality of sports participants. (S8200) Specifically, the player tracking server 2000 may acquire a plurality of image data from the positioning sensor device 200 worn by a plurality of sports participants through the communication module 1100.

In step S8200, the player tracking server 2000 may obtain positioning sensor data transmitted from the positioning sensor devices 200 worn by a plurality of sports participants. The positioning sensor data may include information about a location related to a sports participant, an identifier of the sports participant, and reliability of a sensor-signal.

The player tracking server 2000 may calculate or obtain positioning sensor-based positions of a plurality of sports participants. (S8300) Specifically, the player tracking server 2000 may acquire a plurality of positioning sensor data from the positioning sensor devices 200 worn by a plurality of sports participants through the communication module 1100. In detail, the player tracking server 2000 may obtain the location of each of a plurality of sports participants from information about the location related to the sports participant included in positioning sensor data. In addition, the player tracking server 2000 may obtain information related to speed or acceleration from not only the location but also information about the location related to the sports participant included in positioning sensor data.

Also, the player tracking server 2000 may create a virtual playfield. (S8400) At this time, the virtual playfield may include a plurality of points corresponding to the locations of the plurality of image capture devices and a plurality of moving points corresponding to the locations of the plurality of sports participants acquired in the step S8300.

The player tracking server 2000, based on the arrangement information related to the positions of the plurality of image capture devices 301, 302, 303, 304 obtained in step S8100, a plurality of points corresponding to the plurality of image capture devices. can create

Additionally, the player tracking server 2000 may generate a plurality of moving points corresponding to the plurality of sports participants based on the positions of the plurality of sports participants obtained in step S8300.

In addition, the player tracking server 2000 includes arrangement information related to poses of the plurality of image capture devices 301, 302, 303, and 304 obtained in step S8100 and a plurality of moving images corresponding to a plurality of sports participants. Based on the points, a plurality of virtual lines may be generated.

In the step S8400, the player tracking server 2000 may be provided to calculate relationships between a plurality of image capture devices and a plurality of sports participants.

For example, the player tracking server 2000 determines the relationship between a plurality of fixed points corresponding to a plurality of image capture devices generated in the virtual playfield and a plurality of moving points corresponding to a plurality of sports participants. can be calculated

As another example, the player tracking server 2000 may calculate positional relevance between a plurality of moving points corresponding to a plurality of sports participants created in the virtual playfield.

For another example, the player tracking server 2000 may calculate a relation, such as an angle between a plurality of virtual lines generated in the virtual playfield.

Also, in the step S8400, the player tracking server 2000 may be implemented to predict whether an occlusion event occurs between a plurality of sports participants.

For example, the player tracking server 2000 performs occlusion between a plurality of sports participants based on positions between a plurality of virtual lines generated in the virtual playfield and a plurality of moving points corresponding to the plurality of sports participants. Event occurrence can be predicted.

Also, the player tracking server 2000 may determine which image capture device has generated an occlusion event between a plurality of sports participants.

For example, referring to FIG. 29 again, a first imaginary line VL1 extending from the first fixed point FP1 to the first moving point MP1 and a second moving point ( A second virtual line VL2 extending to MP2 may be created.

At this time, when the first virtual line VL1 and the second virtual line VL2 overlap or the angle between the first virtual line VL1 and the second virtual line VL2 is less than a predetermined angle, The player tracking server 2000 determines or predicts that an occlusion event may occur between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2. can

At this time, the player tracking server 2000 additionally determines that an occlusion event is first fixed between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2. It may be determined or predicted that it may occur from the “first image capture device 301” corresponding to the point FP1.

As another example, the player tracking server 2000 may predict whether or not an occlusion event occurs between a plurality of sports participants based on a position “difference” between a plurality of moving points corresponding to the plurality of sports participants.

For example, referring to FIG. 29 , even if the angles of the first virtual line VL1 and the second virtual line VL2 are the same, the first moving point MP1 and the second moving point MP2 According to the positional difference between the first fixed point FP1, the error between the first sports participant corresponding to the first moving point MP1 and the second sports participant corresponding to the second moving point MP2 The presence or absence of closure may be different.

Accordingly, the player tracking server 2000 may predict whether or not an occlusion event occurs between sports participants by further considering the position difference between moving points corresponding to a plurality of sports participants.

The player tracking server 2000 may create a matching table. (S8500)

In the step S8400, the player tracking server 2000 determines the relationship between the plurality of image capture devices and the plurality of sports participants and the occlusion prediction result between the plurality of image capture devices and the plurality of sports participants. A matching table between sports participants can be created.

For example, a plurality of moving points (FP1, FP2, FP3, FP4) corresponding to a plurality of sports participants ( Virtual lines may be extended for each of a plurality of moving points including MP1, MP2, and MP3, and a table for a prediction result of an occlusion event based on an angle between the virtual lines may be generated.

As another example, the player tracking server 2000 may generate a matching table based on whether or not occlusion is predicted according to a positional difference between a plurality of moving points corresponding to a plurality of sports participants.

As another example, the player tracking server 2000 may generate a matching table based on whether or not occlusion is predicted according to velocity data (or acceleration data) of a plurality of sports participants and pose information of a plurality of image capture devices. .

However, the above information is merely an example, and a matching table may be generated by any suitable method.

The player tracking method, player tracking device, and player tracking system using a plurality of image capture devices according to an embodiment of the present application described above show the position of a sports participant due to an occlusion event, compared to the case of using a single image capture device. The possibility of errors in calculations can be significantly reduced.

See Figures 9 and 31. 31 is a schematic diagram illustrating a method for calculating a location of a sports participant by a tracking method using a plurality of image capture devices according to an embodiment of the present application.

Referring to FIG. 9(b) , when a single image capture device is used, an error may occur between the actual position L2 and the position L2_image obtained from image data according to the vertical movement of the sports participant 1. .

On the other hand, referring to FIG. 31 , when a plurality of image capture devices are used, an error between the actual position (L3) and the position (L3_image1, L3_image2) obtained from image data is corrected according to the vertical movement of the sports participant 1. can do.

For example, correction parameters for positions L3_image1 and L3_image2 obtained from image data may be calculated by considering heights of a plurality of image capture devices and angles to the ground. The actual position L3 of the sports participant 1 can be easily calculated if the calculated correction parameters are applied to the positions L3_image1 and L3_image2 obtained from the image data.

In addition, in the case of using only a single image capture device, if an occlusion event occurs in image data obtained from the image capture device, an alternative method for obtaining an accurate position using only image data is lacking.

On the other hand, in the case of using a plurality of image capture devices, even if an occlusion event occurs in the first image data, the position of the sports participant can be accurately calculated using the second image data in which no occlusion event is detected. .

Also, when a plurality of image capture devices are used, a panoramic view image or an image with increased quality may be generated by matching a plurality of image data acquired from the plurality of image capture devices.

At this time, a plurality of image data may be matched based on references such as a reference point, a reference line, or a reference plane of an overlapping region between the plurality of image data to generate an image of a panoramic view or an image with increased quality.

The player tracking method, player tracking device and player tracking system disclosed in this application can be used to analyze sports players. Specifically, it can be used in all sports analysis fields, such as sports player motion, speed, acceleration, sports player motion, and tactical analysis, which require accurate calculation of the sports player's position.

In addition, although soccer is mainly described in the present application, this is only an example and can be applied to all sports requiring calculation of the position of a sports player.

In addition, since the player tracking method, player tracking device, and player tracking system disclosed in this application can continuously track sports players, they can be used in various fields such as broadcasting related to images and video content related to highlight generation. will be.

The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, etc. illustrated in each embodiment can be combined or modified with respect to other embodiments by those skilled in the art in the field to which the embodiments belong. Therefore, contents related to these combinations and variations should be construed as being included in the scope of the present invention.

In addition, although the embodiment has been described above, this is only an example and does not limit the present invention, and those skilled in the art to the present invention pertain to the above to the extent that does not deviate from the essential characteristics of the present embodiment. It will be appreciated that various modifications and applications not exemplified are possible. That is, each component specifically shown in the embodiment can be implemented by modifying it. And differences related to these modifications and applications should be construed as being included in the scope of the present invention as defined in the appended claims.

Claims (16)

A method for tracking sports participants, the method comprising:
receiving a plurality of sports images from a plurality of cameras installed at different locations around the playfield, each of the plurality of sports images including the playfield and at least a portion of the sports participants;
detecting a target participant from the plurality of sports images;
selecting at least one candidate image from the plurality of sports images according to a detection result of the target participant;
detecting whether occlusion is associated with the target participant from the at least one candidate image;
selecting at least one valid image from the at least one candidate image according to a result of the occlusion detection; and
determining a location of the target participant based on a pixel position corresponding to the target participant in the at least one valid image;
Determining the position of the target participant includes selecting a single image from the at least one valid image based on a first condition related to a position between the target participant and a camera that took the at least one valid image; and
Comprising calculating the location of the target participant based on the single image,
How to track sports players.
According to claim 1,
If the at least one valid image includes a first valid image and a second valid image, determining the location of the target participant comprises:
obtaining a first position of the target participant based on a pixel position corresponding to the target participant in the first valid image;
obtaining a second position of the target participant based on a pixel position corresponding to the target participant in the second valid image; and
Comprising calculating the location of the target participant based on both the first location and the second location.
How to track sports players.
According to claim 1,
Determining the location of the target participant,
Further comprising selecting a single image from the at least one valid image based on the first condition and a second condition related to a lens distortion parameter of a camera that took the at least one valid image.
How to track sports players.
A method for tracking sports participants, the method comprising:
receiving a plurality of sports images from a plurality of cameras installed at different locations around the playfield, each of the plurality of sports images including the playfield and at least a portion of the sports participants;
detecting a target participant from the plurality of sports images;
selecting at least one candidate image from the plurality of sports images according to a detection result of the target participant;
detecting whether occlusion is associated with the target participant from the at least one candidate image;
selecting at least one target image from the at least one candidate image according to a result of the occlusion detection;
When the occlusion is detected in all of the at least one candidate image, determining whether the sports participant occluded or was occluded by another sports participant in the at least one candidate image, and according to the determination select the target image from the at least one candidate image; and
determining a location of the target participant based on a pixel position corresponding to the target participant in the at least one target image;
Determining the position of the target participant includes selecting a single image from the at least one valid image based on a first condition related to a position between the target participant and a camera that took the at least one valid image; and
Comprising calculating the location of the target participant based on the single image,
How to track sports players.
According to claim 4,
wherein the at least one target image is selected from the at least one candidate image in which the sports participant occludes another sports participant in the at least one candidate image.
How to track sports players.
According to claim 4,
If the at least one target image includes a first target image and a second target image, determining the location of the target participant comprises:
obtaining a first location of the target participant based on a pixel position corresponding to the target participant in the first target image;
obtaining a second location of the target participant based on a pixel position corresponding to the target participant in the second target image; and
Comprising calculating the location of the target participant based on both the first location and the second location.
How to track sports players.
According to claim 4,
Determining the location of the target participant,
Further comprising selecting a single image from the at least one target image based on the first condition and a second condition related to a lens distortion parameter of a camera that captured the at least one target image.
How to track sports players.
A method for tracking sports participants, the method comprising:
Obtaining camera arrangement information including positions and poses of a plurality of cameras located around the playfield;
receiving sensor signals from a plurality of positioning sensors located on a plurality of sports participants;
obtaining locations of the plurality of sports participants based on the sensor signals;
Creating a virtual playfield including a plurality of fixed points corresponding to the plurality of cameras and a plurality of moving points corresponding to the plurality of sports participants - the positions of the plurality of fixed points in the virtual playfield are approximately the same as the camera arrangement , where the locations of the plurality of moving points within the virtual playfield are determined based on the locations of the plurality of sports participants;
calculating an association between the plurality of cameras and the plurality of sports participants based on relative positions between the plurality of fixed points and the plurality of moving points;
predicting occlusion of the plurality of sports participants from the plurality of cameras based on angles between a plurality of virtual lines extending from the plurality of fixed points to the plurality of moving points and positions between the plurality of moving points; and
generating a matching table between the plurality of cameras and the sports participant based on the relevance and the occlusion prediction.
How to track sports participants.
In the sports participant tracking device,
A communication module for receiving a plurality of sports images from a plurality of cameras installed at different locations around the playfield; and
A controller analyzing the plurality of sports images received by the communication module;
each of the plurality of sports images includes the playfield and at least a portion of the sports participants;
The controller analyzes the plurality of sports images received by the communication module to detect a target participant from the plurality of sports images, and selects at least one candidate image from the plurality of sports images according to a detection result of the target participant. and detecting whether or not occlusion related to the target participant is present from the at least one candidate image, selecting at least one effective image from the at least one candidate image according to a detection result of the occlusion, and selecting the at least one effective image from the at least one candidate image. determine a location of the target participant based on a pixel position corresponding to the target participant in an effective image of
The controller selects a single image from the at least one valid image based on a first condition related to a position between the target participant and a camera that took the at least one valid image, and based on the single image, the target participant Calculating the position of the target participant by calculating the position of
Sports participant tracking device.
According to claim 9,
The controller,
if the at least one effective image includes a first effective image and a second effective image, a first position of the target participant is obtained based on a pixel position corresponding to the target participant in the first effective image; 2 obtain a second position of the target participant based on a pixel position corresponding to the target participant in an effective image, calculate a position of the target participant based on both the first position and the second position, and to calculate the position of
Sports participant tracking device.
According to claim 9,
The controller,
Selecting a single image from the at least one valid image based on the first condition and a second condition related to a lens distortion parameter of a camera that has taken the at least one valid image.
Sports participant tracking device.
In the sports participant tracking device,
A communication module for receiving a plurality of sports images from a plurality of cameras installed at different locations around the playfield; and
A controller analyzing the plurality of sports images received by the communication module;
each of the plurality of sports images includes the playfield and at least a portion of the sports participants;
The controller detects a target participant from the plurality of sports images, selects at least one candidate image from the plurality of sports images according to a detection result of the target participant, and associates with the target participant from the at least one candidate image. Detect whether or not occlusion is present, select at least one target image from the at least one candidate image according to a result of the occlusion detection, and when the occlusion is detected in all of the at least one candidate image, the sports determine whether a participant has obscured or been obscured by another sports participant in the at least one candidate image, select the target image from the at least one candidate image according to the determination, and select the target image from the at least one candidate image; determine a location of the target participant based on a pixel position corresponding to the target participant in a target image of
The controller selects a single image from the at least one valid image based on a first condition related to a position between the target participant and a camera that took the at least one valid image, and based on the single image, the target participant Calculating the position of the target participant by calculating the position of
Sports participant tracking device.
According to claim 12,
The controller,
selecting, as the at least one target image, the at least one candidate image in which the sports participant occludes another sports participant in the at least one candidate image;
Sports participant tracking device.
According to claim 12,
The controller,
if the at least one target image includes a first target image and a second target image, a first position of the target participant is obtained based on a pixel position corresponding to the target participant in the first target image; 2 obtain a second position of the target participant based on a pixel position corresponding to the target participant in a target image, calculate a position of the target participant based on both the first position and the second position, and the target participant; to determine the location of
Sports participant tracking device.
According to claim 12,
The controller,
Selecting a single image from the at least one target image based on the first condition and a second condition related to a lens distortion parameter of a camera that has taken the at least one target image,
Sports participant tracking device.
In the sports participant tracking device,
a communication module for receiving sensor signals from a plurality of positioning sensors located on a plurality of sports participants;
a controller for acquiring locations of the plurality of sports participants based on sensor signals received by the communication module;
The controller obtains camera arrangement information including positions and poses of a plurality of cameras located around the playfield, and a plurality of fixed points corresponding to the plurality of cameras and a plurality of moving movements corresponding to the plurality of sports participants. Creating a virtual playfield containing points - positions of a plurality of fixed points in the virtual playfield are determined based on the degree of camera arrangement, and positions of a plurality of moving points in the virtual playfield are determined based on the position of the plurality of sports participants. determined based on location; calculates an association between the plurality of cameras and the plurality of sports participants based on relative positions between the plurality of fixed points and the plurality of moving points; Predicting occlusion of the plurality of sports participants in the plurality of cameras based on angles between a plurality of virtual lines extending to the moving point and positions between the plurality of moving points, and based on the relevance and the occlusion prediction generating a matching table between the plurality of cameras and the sports participant based on the
Sports participant tracking device.

Images