WO2023089381A1

WO2023089381A1 - The method and system of automatic continuous cameras recalibration with automatic video verification of the event, especially for sports games

Info

Publication number: WO2023089381A1
Application number: PCT/IB2022/054962
Authority: WO
Inventors: Zbigniew KORCZAK; Robert Sitnik
Original assignee: Smart Tracking Sp. Z O.O.
Priority date: 2021-11-19
Filing date: 2022-05-26
Publication date: 2023-05-25
Also published as: PL439581A1

Abstract

The subject of the invention is a method and a system for automatic continuous cameras recalibration along with automatic video verification, especially for sports games such as volleyball, tennis and others. The aim of our invention is to build a new, more accurate system for video verification in sport and judge support, which will enable more accurate and precise measurement of a given event (ball, block, line crossing, etc.) with the use of continuous auto-calibration - system recalibration during operation, the system being characterized by will be reinforced resistance to shocks and hits of the ball / player, which will not affect its functioning and quality of measurements, including verification video. In addition, the system is to enable proper operation even with the deformation of the ball reaching 50% of its shape-size. The system according to the invention will be used as a vision system supporting the work of a referee especially volleyball referee.

Description

The method and system of automatic continuous cameras recallbration with automatic video verification of the event, especially for sports games.

The subject of the invention is a method and a system for automatic continuous cameras recalibration along with automatic video verification, especially for sports games such as volleyball, tennis and others.

Multi-camera vision or optical systems are increasingly used in supporting the work of judges in the event of sports competitions. In addition to the well-known Hawk-Eye system for determining the trajectory of the ball during tennis matches and for determining the position of the ball in soccer competitions, there are many different systems that are used to support the referee, such as GLT in soccer, or preview systems (replay or video verification ) from cameras in the volleyball. These systems represent different approaches to assisting the judge. The Hawk-Eye is a well-calibrated (each time is calibrated to run) multi-camera system that can estimate the position of the ball in each set of frames collected. However, system calibration is always performed offline - both intemal camera calibration and a camera position estimation in the global coordinate system. In such case, the displacement of any camera, e.g. caused by the operator or hitting a ball, may deteriorate the accuracy of the system or prevent its further efficient operation.

In addition, the well-known inout system ( Hawk-Eye ) - it is used to perform video replays, there is a version of the system with 10 cameras placed around the playing field, the system triangulates the positions and predicts what may happen on the playing field with the ball, tracking it frame by frame to determine a single ball trajectory; In the Hawk-Eye system, the central computer calculates the data and provides it with an accuracy of 5mm; the repetition count is less than 10 seconds; a system used in tennis since 2002 (https://www.youtube.com/watch?v=6wjhSR5Dcyk).

There are also known systems in which ball tracking is used to generate statistics (eg

PlayFul Vision ), but they are by no means a competition due to the insufficient accuracy for the purpose of refereeing, resulting from the technology used, ie doppler echo measurement.

Competitive companies have a standard system of video verification, which consists in video recording of events on the playing field and, if the referee's decision is questioned, playing a video of the disputed situation. The judges, while watching the repetition, make the decision to award a point.

The HawkEye company additionally has one function of automatic ball detection - whether the ball after service, attack, block, puncture to the other side of the playing field hit the playing field or not.

TDS International offers HD video verification (where the set consists of only 3 HD cameras and 13 SD quality cameras), and Data Project and HawkEye companies are below the HD standard. From the American patent No. US8734214B2 there Is known a system, method and computer program product for simulating movement of a projectile in a virtual environment Primary data comprising a plurality of sets of position values and a pluraity of time values for a projectile is received, with each time value in the primary data being associated with one of the sets of position values. The primary data is processed to generate secondary data that represents at least two consecutive sets of positions. Each set of positions Includes a start position and an end position for the projectile, and the secondary data further represents a velocity of the projectile associated with each start position. The secondary data is used to simulate movement of the projectile in the virtual environment.

US2019287310A1 discloses a method includes receiving two-dimensional video streams from a plurality of cameras, the two-dimensional video streams Including multiple angles of a sporting event The method further Includes determining boundaries of the sporting event from the two- dimensional video streams. The method further includes identifying a location of a sporting object during the sporting event The method further includes identifying one or more players In the sporting event The method further includes Identifying poses of each of the one or more players during the sporting event. The method further includes generating a three-dimensional model of the sporting event based on the boundaries of the sporting event, the location of the sporting object during the sporting event, and the poses of each of the one or more players during the sporting event The method further Includes generating a simulation of the three-dimensional model

From the International patent No. WO2019155271A1 an interactive sports simulation system and a method of training the user are known. The system Includes a feedback device configured to allow the user to perform a sporting activity, including vibration motors and a handte. Vibration motors Include linear resonant actuators, a first rotating mass eccentric motor and a second rotating mass eccentric motor. The system further Includes a first processing device configured to cooperate with the vibration motors and an end user device. The end user device Is configured to detect a user- generated sport activity and transmit Afferent signals corresponding to different tactile feedback patterns to the first processing device based on the user sport activity and vibration motors configured to generate different tactile feedback patterns. The system further Includes a foot detection device and a two-dimensional tracking device, including light sensitive sensors on rods and laser emitters along the line of sight of the photosensitive sensors. Laser emitters configured to emit laser beams on photosensitive sensors to detect user presence and a two-dimensional location in a play area. The system further Includes at least one second processing device located In the foot detection device and at least one third processing device housed in a two-dimensional tracking device configured to cooperate with laser emitters and photosensitive sensors.

A device for simulating a sports for referee is known from the Chinese patent No. CN107982900A. The simulation device Includes a control mechanism, detection mechanisms, and a voice reproduction mechanism. The control mechanism is internally equipped with sports rules for various sporting events; the detection mechanisms are communicatively linked to the control mechanism and are distributed over the sports equipment and sports grounds and are used to detect information about the sports equipment or sports playing field and to transmit the information to the control mechanism; the voice reproducing mechanism is in communication with the control mechanism and is used to receive the instructions sent by the control mechanism and reproduce the judgment results according to the instructions; the control mechanism receives the information detected by the detecting mechanisms and sends instructions after being assessed according to the rules of sport. With the Sport Referee Simulator, you can simulate the referee's judgment while achieving fairness and equality, and the referee workload is greatly relieved.

From the international patent No. W02014148927A1 there is known a system for video verification of events on the playing field with the use of cameras tracking the movement of the ball, a device for recording the movement of the ball with the ability to reproduce on demand, and a display for displaying a fragment or the whole event on demand. The system is characterized by the fact that it consists of at least one group of cameras observing the following elements of the event: ball, players, lines or any combination of these elements, possibly equipped with cameras (K1, K2, Kn ) of the respective groups with autonomous recorders, and they are also connected to a central module (MC) equipped with at least a visualization device (VM) and a control device (CM). The central module (MC) allows access to event images for verification in less than 10 seconds.

Austrian patent AU2007219710A1 discloses a method and device for determining the parameters of contact between bodies during activities, in particular in the context of sports, games, recreation or entertainment, such as bouncing a tennis ball or contacting a cricket or baseball ball.

A digital camera aimed at the target area captures image frames containing infrared image data in the spectral range during the operation. The means for comparing the image data for a plurality of staggered frames with the reference image data records each localized infrared emission appearing in at least one event frame, the localized infrared emission indicating that the excitation was generated by a contact that occurred in at least one event frame, the target area and which has generated heat, such as friction between the bodies or deformation of at least one of the bodies involved in the contact.

The picture data for the at least one event frame is parsed to determine the at least one contact parameter, and the picture data for the event frame is recorded. Visual display means displaying incident frames, including image data in the IR spectrum, presenting the IR data as a visible representation of the scene captured for the incident frame to allow visual control of the activity in the target area over time at least one event frame that gave rise to the contact producing the infrared emission.

The aim of our invention is to build a new, more accurate system for video verification in sport and judge support, which will enable more accurate and precise measurement of a given event (ball, block, line crossing, etc.) with the use of continuous auto-calibration - system recalibration during operation, the system being characterized by will be reinforced resistance to shocks and hits of the ball / player, which will not affect its functioning and quality of measurements, including verification video. In addition, the system is to enable proper operation even with the deformation of the ball reaching 50% of its shape-size.

The method according to the invention characterized in that, in the first step signals (S) in the form of data ( D) from the cameras (K), preferably from 12 cameras (K), are sent to the main computer

(KG) and frame by frame is recorded in the memory (Pa), and then calibrated by setting their position in relation to the line position (L) on playing field (B) in the image (O) by determining their occupancy in pixels for each of the frames read for a given camera (K); such auto- calibration is performed for each of the cameras (K), preferably separately or simultaneously, at the beginning of operation - after activating the system and / or cameras (K), and each time a shock is detected on the shock- accelerometer sensor (C) for a given camera (K) and / or shifts of line coordinates (L) on the playing field (B) in the image (O) from each of the cameras (K), the auto- calibration is preferably performed by the software ( Op ) - the algorithm implemented in the main computer (KG) and / or local computer (KL), while at the same time, changing the position of the line for any of the cameras (K) in the image (O) results in an immediate automatic recalibration - auto-calibration of this camera (K) to a new position of the line (L), wherein calibration of each camera (K) is taking place cyclically in the coupling feedback, preferably every 5 minutes while the system is operating; and both the internal calibration of the cameras (K) and the estimation of position and orientation of each camera (K) in the global coordinate system are performed as calibration; wherein the signals (S) - data (D) from the cameras (K) in the video form are in image quality (O) - video at least FULL HD, which means at least contain 1080 lines per image (0); and in the second processing step, the previously obtained data (D) is preprocessed from the image (0), which means the position of the ball center (SP) is determined for each frame on the images (O) from cameras (K) on local computers (KL), which means on computers next to the cameras (K); for each of the images (O) is determined, and then the detected center (SP) is sent in the form of data to (KG), and there, in each frame, the position of the ball (P) in 3D is additionally determined by overlapping the data (D) in time of the at least two cameras (K) preferably of the entire system it is 12 cameras (K); while in the third decision-making step, for the entire sequence of received data (D), the trajectories of the ball (P) are determined in the coordinate system of the playing field (B) and on its basis individual events (Z) related to the ball (P) on the playing field (B) are detected such as: -In / out - whether the ball (P) is inside the playing field (B) or out,

- block - whether the ball (P) changed its trajectory or not,

- putting up a player's hand - whether the ball (P) rebounded from the plane of the playing field (B) or from another, where the point and angle of the ball's rebound (P) are additionally taken into account; and the system displays the position for each of the cameras ( K) by visualizing the pixel occupancy in the frame in the image (O) by the ball (P) and determining the center of the ball (P), wherein system additionally takes into account and detects ball deformations (P) up to a maximum of 50% of the shape - size of the ball (P), by changing the distribution of pixels in relation to the center of the ball (P), while the ball (P) is presented on the local computer ( KL) and software ( Op ), as a point or sphere, while for the proper operation of the system, at least two cameras are needed (K) placed around the playing field (B) to triangulate the position of the ball (P) and the event (Z); such recorded data of the ball (DP) processed in this way, that is additionally used to build a model of the ball trajectory (P), and to predict and thus anticipate the trajectory of the further potential trajectory of the ball (P) and / or to perform a 3D simulation - video simulation ; preferably with an accuracy of 2 mm with respect to the position of the center-center of the ball (P).

The auto-calibration - calibration is performed by means of software ( Op ), i.e. by means of an instruction - an algorithm that preferably reads the previous correct line position and applies a spatial correction to the offset.

At least one user, preferably to the match referee, is displayed after the event (Z) on the mobile device with the message (Kom) - preferably a smartwatch.

Fact that the artificial intelligence contained in the software ( Op ) is additionally used to recognize elements on the playing field (B), such as lines (L).

In order to obtain an accuracy of 2 mm in the measurement of the position of the ball (P) in the event (Z), the dimension of the ball (P) in the image (O) must be at least 20x20 pixels.

The system according to the invention characterized in that, system contains cooperating, wired and / or wirelessly connected and exchanging signals (S) • data in two directions: cameras (K) preferably 12 cameras (K) arranged preferably around-around the playing field (B), connected to a central computer - the main computer (KG) processing and saving in memory (Pa) the image (O), and the recorded material is analyzed and processed by the software (Op) in order to determine the path

- the flight of the ball and its prediction-anticipation, wherein system after the event (Z), preferably when is so-called challenge , thanks to the software

(Op), will automatically make a decision and display the message (Kom) on the display (W) to the operator - the referee and / or shows a video replay for the referee's independent decision, while the playing field lines create constant coordinates for the calibration - auto-calibration of each of the cameras (K), wherein all system components are connected to the power supply.

Multi-camera vision or optical systems can be used as cameras (K). Each of the cameras (K) additionally has its own separate local computer (KL) and / or a microcircuit - computing chip with which it is connected and exchanges data bi-directionally, while the local computer (KL) transmits bi-directional data-signals (S) to the main computer ( KG).

It additionally comprises sensors (C) preferably mounted on and / or on the cameras.

As sensors (C) are used: motion sensor, shock sensor, accelerometer.

In the case of the so-called ” Video verification" data (D) from cameras (K) in the visual form are transferred as two-dimensional - images (O) to the main computer (KG) in the image-video quality of at least FULL HD, which means, they contains at least 1080 lines per image.

In the case of the so-called " Challange " data (D) from cameras (K) in the visual form are analyzed as two-dimensional data (D) - images (O) in local computers (KL) in the form of 2D, and then transferred to the main computer (KG), where are analyzed in 3D form.

Memory (Pa) is located locally and / or remotely on the server and / or in the cloud.

System -main computer (KG) illustrates the ball (P) as a point or a sphere.

Software (Op) is located-implemented in the main computer (KG) and local computers (KL), while system is a distributed system and can operate modularly, and preferably some of it can operate independently.

Algorithms and / or decision clouds and / or artificial intelligence are used as software (Op), preferably - neural networks, convolutional neural networks, classification algorithms and / or self-learning mechanisms; the artificial intelligence is additionally used to more accurately recognize elements on the playing field

(B) like lines (L).

System additionally visualizes a given situation - event (Z) on the basis of processed ball data (DP) in the form of a 3D animation, which is displayed on a large screen and / or on viewers' TVs.

System is also a video verification system - it is for replaying recorded video.

System is also a video verification system - it is used to play the replay of recorded video with augmented reality - called AR , including additional data / information for fans / referee .

System enables the playback of stored data for the fan user in the so-called on demand - "on demand " and allows you to watch, at any time, the indicated repetition of an event (Z) or challenge, as a simulation from various angles, allowing the user to freely control it in 3D.

Details are decisive for winning or losing in a sports competition at the highest level, and it is at stake in acquiring sponsors and large financial bonuses. The entire world of sport strives to make refereeing as little as possible from human error, which results in the most objective result. This need is answered by the solution according to the invention, the aim of which is to develop a vision system for the correction of errors most often made by volleyball referees, which will maximally objectify the way of assessing the most frequently occurring disputable situations during volleyball games. The system, by means of 12 cameras placed around the playing field, connected to a central computer processing the image, will record events on the playing field during the match. If the referee's dedsion is questioned, the recorded material will be analyzed by the software in order to determine the ball's trajectory and the system will automatically decide whether there has been an error and which team to award a point to.

In addition, the system will visualize the situation in the form of a 3D animation, which will be displayed on the large screen and on viewers' TVs, similar to the "challenge" in tennis or checking whether the ball has crossed the goal line in football.

The system will automatically (without the participation of a judge) analyze three situations that are the subject of the most common doubts and controversies:

- whether the ball on the serve, attack, block, hit to the other side of the playing field is in the playing field of play or not ("in / out"),

- whether the player has managed to put his hand under the ball so that it does not come into contact with the playing field of play ("Floor Touch"),

- whether or not the ball has touched the fingers, edge of the hand or forearm of the blocking player

(s) ("Block Touch").

In the three most difficult situations described above, even though the existing tools for video verification are used, the judges make the most statistically (approx. 80%) errors (according to PZPS data). These functionalities will distinguish our system from the video verification systems available on the market.

The system will also be a video verification system - a system used to play a video replay, where the judge watching the replay makes a decision in situations such as:

- whether the ball had contact with the antenna,

- has the player touched the net,

- whether the player made the mistake of stepping on the line or crossing it.

-the position of the libero in play when the ball ends in an attack after his display.

As in competing systems, the video replay will be able to be displayed on a large screen and TV broadcast

SYSTEM COMPETITIVE ADVANTAGE

The system according to the invention will have the following competitive advantages:

- automatic verdict for the three most important contentious playing field situations; competition allows only manual verification of these situations (except for one of them - Hawk-Eye , see above);

- speed of giving the verdict approx. 20 seconds; in the case of competitions, the time to consider the

"challenge" on the basis of the standard video verification is 2-3 minutes; - full system mobility and low installation costs; unfolding / folding up to 2h; competitive systems require the room to be prepared even a day before the match, and in the case of Hawk-Eye , the initial configuration and assembly (on a tennis court, which is smaller and less complicated than a volleyball playing field) takes about 3-4 days ; This translates into convenience for the organizers and lower service costs on the part of our company (less man-hours of service and the possibility of multiple use of the same set of equipment in different rooms, even on the same weekend, for example);

- the ability to present synchronized image from many cameras at the same time (the competition shows the image from only one camera);

- automatic recalibration of cameras; this may be needed, for example, in the event of a ball hitting the camera; in the case of competition, it is done manually, which takes time and may cause errors in further operation;

- the cost of the hardware set is much lower.

The system according to the invention will be used as a vision system supporting the work of a volleyball referee.

The main technical problems that the invention solves are: a) 3D detection - events / balls and making a decision, b) auto- calibration in feedback operating in the system, operating separately for each camera, i.e. independently, as well as for the entire system of 12 cameras, c) taking into account the deformation of the ball when hitting an object (line, hands, block, net), up to 50% of the deformation of the shape of the ball; and related implemented hardware / software solutions, including the use of innovative algorithms for data recognition and processing.

Moreover, 3D detection ensures accuracy regardless of the direction of observation, where with 2D detection, the accuracy depends on the direction of observation.

Moreover, the system according to the invention is a measuring system where we make measurements and compare the value for which the measurement deviates from the given value.

The subject of the invention is shown in the examples of the drawing in which:

Fig. 1 - shows the system according to the invention in a top view with an arrangement of cameras,

Fig. 2 - shows a diagram of the arrangement of cameras for a half of the playing field,

Fig. 3 - shows an example of determining the position of the ball, the center of the ball and the deformation on the bounce off the ground. Example 1 (System. 12 cameras around the playing field, main computer and local computers by each camera, distributed system; fig. 1)

The system includes cooperating, wired and exchanging signals (S) - bi-directional data:

12 cameras (K) positioned around the playing field (B), connected to a central computer - the main computer (KG) processing and saving to memory (Pa) the image (O), and the recorded material is analyzed and processed by the software (Op) in order to determine the path- the flight of the ball and its prediction-anticipation.

After the event (Z), that is, when the so-called challange , thanks to the software (Op), the system automatically makes decisions and displays the message (Kom) on the display (W) to the operator - the referee and / or shows a video replay for the referee's own decision.

The playing field lines create constant coordinates for the calibration - auto- calibration of each camera (K), all system components are connected to the power supply.

Multi-camera vision or optical systems can be used as cameras (K).

Each of the cameras (K) additionally has its own separate local computer (KL) and with which it is connected and exchanges data bidirectionally, while the local computer (KL) transmits bidirectionally data-signals (S) to the main computer (KG).

Additionally, it includes sensors (C) mounted on / next to the cameras.

As sensors (C) are used: motion sensor, shock sensor, accelerometer.

In the case of the so-called " Video verification" data (D) from cameras (K) in the visual form are transferred as two-dimensional - images (O) to the main computer (KG) in the image-video quality of at least FULL HD, that is, they contain at least 1080 lines per image.

In the case of the so-called "Challange" data (D) from cameras (K) in the visual form are analyzed as two-dimensional data (D) - images (O) in local computers (KL) in the form of 2D, and then transferred to the main computer (KG), where are analyzed in 3D.

Memory (Pa) is located locally or remotely on the server (possibly in the cloud).

System · the main computer (KG) shows the ball (P) as a point or a sphere.

The software (Op) is located-implemented in the main computer (KG) and local computers (KL)

The system is a distributed system and can operate modularly, and some of it can operate independently.

Algorithms or decision clouds (or self-learning mechanisms) are used as software (Op).

Artificial intelligence is additionally used to more accurately recognize elements on the playing field (B), such as lines (L).

In addition, the system will visualize a given situation - event (Z) on the basis of processed ball data (DP) in the form of a 3D animation, which is displayed on the large screen / TV viewers.

The system is also a video verification system - this is for replaying recorded video. The system is also a video verification system - it is used to play the replay of the recorded video with augmented reality - called AR - Augmented Reality, including additional data / information for fans / referee.

The system enables the playback of stored data for the fan user in the so-called - "on demand " and allows you to watch, at any time, the indicated repetition of an event (Z) or challenge, as a simulation from various angles, allowing the user to freely control it in 3D.

Processing steps - general system operation:

1) Calibration of individual cameras (K).

2) Set-up of the system on the playing field (B) + combination of power supply and signals (S).

3) Cross-calibration of cameras (K) in the playing field (B) pattern, where the playing field (B) represents the plane (XY).

4) Permanent recording of a sequence from each camera (K) to a local computer (KL) next to the camera (K)

5) In the event that the so-called challenge: a. Choosing what type of challenge it is. b. Processing the images (O) of the last recorded action on local computers (KL) and sending the detected center of the ball (SP) of the chosen ball (P) (in image coordinates) to the main computer (KG). c Convert 2D centers to the position of the ball center (P) in a 3D system related to the playing field (B). d. Determining the trajectory of the ball, taking into account its deformation when bouncing off obstacles.

Moreover, we assume that the size of the ball (P) must be at least 20x20 pixels to obtain an accuracy of 2mm.

Techniques for processing and recognizing 2D and 3D image information in real time:

In local computers (KL), based on the shape of the detected ball, its center is corrected, taking into account both its potential deformation and the eccentricity error.

Then, the centers of the ball (P) in the image coordinates (0) are analyzed and the center of the ball (P) is determined in the 3D coordinates related to the playing field (B).

System:

System components: Multicamera vision or optical systems - cameras (K)

Basic variant: 12 cameras, server computer, stands, cables, cameras, lenses, UPS.

The software is in the main (KG) and local (KL) computer.

Example 2 (general method - event recognition)

In the first stage, signals (S) in the form of data (D) from cameras (K), from 12 cameras (K), are sent to the main computer (KG) and frame by frame is recorded in the memory (Pa), and then calibrated them by determining their position in relation to the position of the line (L) of the playing field (B) in the image (O) by determining their occupancy in pixels for each of the frames read for a given camera (K); such auto- calibration is performed for each of the cameras (K) separately, at the beginning of operation - after activating the system and / or cameras (K), and each time when a shock is detected on the shock-accelerometr sensor (C) for a given camera (K) or shifting the coordinates of the line (L) on the playing field (B) in the image (O) from each of the cameras (K).

Autocalibration is performed by the software (Op) - an algorithm implemented in the main computer

(KG) and the local computer (KL), at the same time, changing the position of the line for any of the cameras (K) in the image (O) results in an immediate automatic recalibration - auto-calibration of this camera (K) to a new position of the line (L), with the calibration of each camera (K) taking place cyclically in the coupling every 5 minutes while the system is running.

As calibration, both the internal calibration of the cameras (K) and the estimation of position and orientation of each camera (K) in the global coordinate system are performed.

Wherein the signals (S) - data (D) from the cameras (K) in the video form are in the image quality (0) - the video at least FULL HD, that is, at least contain 1080 lines per image (O);

In the second processing step the previously obtained data (D) is preprocessed from the image

(O), which means the position of the ball center (SP) is determined for each frame on the images (O) from cameras (K) on local computers (KL), which means on computers next to the cameras (K); for each of the images (0) is determined, and then the detected center (SP) is sent in the form of data to (KG), and there, in each frame, the position of the ball (P) in 3D is additionally determined by overlapping the data (D) in time of the at least two cameras (K) preferably of the entire system it is 12 cameras ( K);

In the third step of decision making, for the entire sequence of received data (D), the trajectories of the ball (P) are determined in the coordinate system of the playing field (B) and on its basis individual events (Z) related to the ball (P) on the playing field (B) are detected such as:

-In / out · whether the ball (P) is inside the playing field (B) or out,

- block - whether the ball (P) changed its trajectory or not.

- putting up a player's hand - whether the ball (P) rebounded from the plane of the playing field (B) or from another, where the point and angle of the ball's rebound (P) are additionally taken into account; and the system displays the position for each of the cameras (K) by visualizing the pixel occupancy in the frame in the image (O) by the ball (P) and determining the center of the ball (P), wherein system additionally takes into account and detects ball deformations (P) up to a maximum of

50% of the shape - size of the ball (P), by changing the distribution of pixels in relation to the center of the ball (P), while the ball (P) is presented on the local computer (KL) and software (Op), as a point or sphere. For the proper operation of the system, at least two cameras are needed (K) placed around the playing field (B) to triangulate the position of the ball (P) and the event (Z); such recorded data of the ball (DP) processed in this way, that is additionally used to build a model of the ball trajectory (P), and to predict and thus anticipate the trajectory of the further potential trajectory of the ball (P) and / or to perform a 3D simulation - video simulation ; with an accuracy of 2 mm with respect to the position of the center-center of the ball (P).

Auto-calibration - calibration is performed by means of software (Op ), i.e. by means of an instruction - an algorithm that reads the previous correct line position and applies a spatial correction to the offset. The message (Kom) is displayed to the match referee, after the event (Z), on the mobile device - a smartwatch.

Fact that the artificial intelligence contained in the software (Op) is additionally used to recognize elements on the playing field (B), such as lines (L).

In order to obtain an accuracy of 2 mm in the measurement of the position of the ball (P) in the event

(Z), the dimension of the ball (P) in the image (O) must be at least 20x20 pixels.

Example 3 (event recognition method ("in / out" )

- Is the ball on the serve, attack, block, punch to the other side of the playing field in the playing field of play or not ("in / out"),

Based on the calibration of the camera system (K) around the playing field (B), the 3D positions and the orientation of the cameras (K) in the playing field layout (B) are known. The positions of the line

(L) of the playing field (B) in the camera coordinate system (K) are known. The 3D position of the ball

(P) during the game is also known and its model of deformation during contact with the ground is known, in particular, its contact area with the surface of the playing field (B) is known depending on the speed and angle of impact.

The decision that the ball (P) is out of the box is made automatically by considering the following situations:

1) the ball (P) did not collide with the ground but is out of bounds (B),

2) the ball (P) has collided with the ground near the edge of the playing field (B) and a thorough analysis is made of whether the contact area of the ball with the playing field (B) touches the line (L) or is outside it.

Situation 2) is more complex and is analyzed as follows: Input data : internal calibration of cameras, position and orientation of cameras (K) in the playing field layout (B), position of the line (L) in the playing field layout (B).

The method of signal analysis when it is necessary to verify the "in / out" situation:

1. The position of the center of the ball (x, y, z) is analyzed over a dozen frames before and after bouncing the ball (P).

2. On this basis, the angle of the ball hitting the plane of the playing field and the angle of this hitting are determined.

3. From the model of ball deformation, its contact area with the playing field is determined (B).

4. The overlap between the playing field line and the contact surface of the ball with the playing field is analyzed (B).

5. In the absence of a contact, "out" is reported, otherwise "in" is reported.

Example 4 (how to recognize an event ("Floor touch " )

- Has the player managed to put his hand under the ball so that it does not come into contact with the playing field of play ("Floor touch"),

Based on the calibration of the camera system (K) around the playing field (B), the 3D positions and the orientation of the cameras (K) in the playing field layout (B) are known. The plane / surface of the playing field (B) in the camera coordinate system (K) is also known. The 3D position of the ball (P) during the game is also known.

The decision that the ball (P) has rebounded the surface of the playing field (B) or completely or partially from a player's hand is made automatically on the basis of an analysis of the following situation:

1) the ball has hit the surface of the playing field (B) and for each impact there is a detailed analysis of whether the player has interfered with the reflection.

Situation 1) is complex and is analyzed as follows:

Input data : internal calibration of cameras (K), position and orientation of cameras in the playing field layout (B), position of the plane / surface of the playing field (B) in the 3D camera system.

The method of signal analysis when it is necessary to verify the situation of " Floor touch

1. The position of the center of the ball (x, y, z) is analyzed over a dozen frames before and after the bounce.

2. On this basis, the starting and ending angle of the ball rebound from the plane of the playing field is determined.

3. From the model of ball deformation, the height of the plane from which the ball bounces is determined. 4. If the designated angles are equal and the reflection plane coincides with the surface of the playing field, then there was no "hand over", otherwise it did.

Example 5 (general event recognition method ("Block touch " )

- Whether or not the ball has touched the blocking player's fingers, edge of hand or forearm ("Block touch ").

Based on the calibration of the camera system (K) around the playing field (B), the 3D positions and the orientation of the cameras (K) in the playing field layout (B) are known. The 3D position of the ball during the game is also known. The position of the grid plane in the 3D camera system is known.

The decision that there was a "Block touch" is made automatically on the basis of the analysis of the following situation:

1) the ball changed or not its trajectory passing over the net; for each collision there is a detailed analysis of whether the trajectory has changed and whether the net or the player has been touched.

Situation 1) is complex and is analyzed as follows:

Input data : internal calibration of cameras, position and orientation of cameras (K) in the playing field layout (B), position of the plane / surface of the playing field in the 3D camera system, position of the grid in the camera system.

The method of signal analysis when it is necessary to verify the " Block touch " situation:

1. The position of the center of the ball (x, y, z) is analyzed over a dozen frames before and after crossing the grid plane.

2. On this basis, the trajectory of the ball is determined and its continuity is analyzed.

3. If the continuity of the trajectory is disturbed, the place of disturbance in the coordinates of the playing field is analyzed.

4. In the vicinity of the disorder, the presence of players' hands is analyzed. If the hands are in contact with the ball, a "Block touch" is reported .

5. Otherwise, there was no "Block touch".

List of elements:

S. signals D. data K. camera KG. main computer KL. local computer O. image Bye. memory Op. software C. sensors Kom. message

P. ball

Sp. the detected center of the ball Z. event

Dp . ball data

B. playing field L lines

Pa- memory

Claims

Patent claims

1. Method of automatic continuous recalibration of cameras with automatic video verification of the event, especially for sports games, using initial calibration and continuous analysis, recognition and tracking of fixed points on the playing field and the ball, including data processing, characterized in that in the first step signals (S) in the form of data (D) from the cameras (K), preferably from 12 cameras (K), are sent to the main computer (KG) and frame by frame is recorded in the memory (Pa), and then calibrated by setting their position in relation to the line position (L) on playing field (B) in the image (O) by determining their occupancy in pixels for each of the frames read for a given camera (K); such auto- calibration is performed for each of the cameras (K), preferably separately or simultaneously, at the beginning of operation - after activating the system and / or cameras (K), and each time a shock is detected on the shock-accelerometer sensor (C) for a given camera (K) and / or shifts of line coordinates (L) on the playing field (B) in the image (O) from each of the cameras

(K), the auto- calibration is preferably performed by the software ( Op ) - the algorithm implemented in the main computer (KG) and / or local computer (KL), while at the same time, changing the position of the line for any of the cameras (K) in the image (O) results in an immediate automatic recalibration - auto calibration of this camera (K) to a new position of the line (L), wherein calibration oF each camera (K) is taking place cyclically in the coupling feedback, preferably every 5 minutes while the system is operating; and both the internal calibration of the cameras (K) and the Block touch of position and orientation of each camera (K) in the global coordinate system are performed as calibration; wherein the signals (S) - data (D) From the cameras (K) in the video Form are in image quality (0) - video at least FULL HD, which means at least contain 1080 lines per image (0); and in the second processing step, the previously obtained data (D) is preprocessed from the image (O), which means the position of the ball center (SP) is determined for each frame on the images (O) from cameras (K) on local computers (KL), which means on computers next to the cameras (K); for each of the images (O) is determined, and then the detected center (SP) is sent in the form of data to (KG), and there, in each frame, the position of the ball (P) in 3D is additionally determined by overlapping the data (D) in time of the at least two cameras (K) preferably of the entire system it is 12 cameras (K); while in the third decision-making step, for the entire sequence of received data (D), the trajectories of the ball (P) are determined in the coordinate system of the playing field (B) and on its basis individual events (Z) related to the ball (P) on the playing field (B) are detected such as: -In / out - whether the ball (P) is inside the playing field (B) or out,

- block - whether the ball (P) changed its trajectory or not,

50% of the shape - size of the ball (P), by changing the distribution of pixels in relation to the center of the ball (P), while the ball (P) is presented on the local computer ( KL) and software ( Op ), as a point or sphere. while for the proper operation of the system, at least two cameras are needed (K) placed around the playing field (B) to triangulate the position of the ball (P) and the event (Z); such recorded data of the ball (DP) processed in this way, that is additionally used to build a model of the ball trajectory (P), and to predict and thus anticipate the trajectory of the further potential trajectory of the ball (P) and / or to perform a 3D simulation - video simulation ; preferably with an accuracy of 2 mm with respect to the position of the center-center of the ball (P).

2. The method according to claim 1, characterized in that, the auto-calibration - calibration is performed by means of software ( Op ), i.e. by means of an instruction - an algorithm that preferably reads the previous correct line position and applies a spatial correction to the offset.

3. The method according to claim 1, characterized In that, the at least one user, preferably to the match referee, is displayed after the event (Z) on the mobile device with the message (Kom) - preferably a smartwatch.

4. The method according to claim 1, characterized In that, the fact that the artificial intelligence contained in the software ( Op ) is additionally used to recognize elements on the playing field (B), such as lines (L).

5. The method according to claim 1, characterized in that, in order to obtain an accuracy of 2 mm in the measurement of the position of the ball (P) in the event (Z), the dimension of the ball (P) in the image (0) must be at least 20x20 pixels.

6. System of automatic continuous recalibration of cameras with automatic video verification of the event, especially for sports games, using: cameras, at least one computer characterized in that, system contains cooperating, wired and / or wirelessly connected and exchanging signals (S) - data in two directions: cameras (K) preferably 12 cameras (K) arranged preferably around-around the playing field (B), connected to a central computer - the main computer (KG) processing and saving in memory (Pa) the image (0), and the recorded material is analyzed and processed by the software (Op) in order to determine the path - the flight of the ball and its prediction-anticipation, wherein system after the event (Z), preferably when is so-called challange , thanks to the software

7. System according to claim 6, characterized in that, multi-camera vision or optical systems can be used as cameras (K).

8. System according to daim 7, characterized in that, each of the cameras (K) additionally has its own separate local computer (KL) and / or a microcircuit -computing chip with which it is connected and exchanges data bi-directionally, while the local computer (KL) transmits bi-directional data-signals

(S) to the main computer ( KG).

9. System according to daim 6, characterized In that, it additionally comprises sensors (C) preferably mounted on and / or on the cameras.

10. System according to claim 6, characterized in that, as sensors (C) are used: motion sensor, shock sensor, accelerometer.

11. System according to claim 6, characterized in that, in the case of the so-called " Video verification'¹ data (D) from cameras (K) in the visual form are transferred as two-dimensional - images (O) to the main computer (KG) in the image-video quality of at least FULL HD, which means, they contains at least 1080 lines per image.

12. System according to claim 6, characterized in that, in the case of the so-called " Challange " data

(D) from cameras (K) in the visual form are analyzed as two-dimensional data (D) - images (O) in local computers ( KL) in the form of 2D, and then transferred to the main computer (KG), where are analyzed in 3D form.

13. System according to claim 6, characterized In that, the memory (Pa) is located locally and / or remotely on the server and / or in the cloud.

14. System according to claim 6, characterized In that, the system -main computer (KG) illustrates the ball (P) as a point or a sphere.

15. System according to claim 6, characterized in that, the software (Op) is located-implemented in the main computer (KG) and local computers (KL), while system is a distributed system and can operate modularly, and preferably some of it can operate independently.

16. System according to claim 6, characterized in that, algorithms and / or decision clouds and / or artificial intelligence are used as software (Op), preferably - neural networks, convolutional neural networks, classification algorithms and / or self-learning mechanisms; the artificial intelligence is additionally used to more accurately recognize elements on the playing field (B) like lines (L).

17. System according to claim 6, characterized in that, the system additionally visualizes a given situation - event (Z) on the basis of processed ball data (DP) in the form of a 3D animation, which is displayed on a large screen and / or on viewers' TVs.

18. System according to claim 6, characterized in that, the system is also a video verification system - it is for replaying recorded video.

19. System according to claim 6, characterized In that, the system is also a video verification system - it is used to play the replay of recorded video with augmented reality - called AR , including additional data / information for fans / referee .

20. System according to claim 6, characterized in that, the system enables the playback of stored data for the fan user in the so-called on demand - "on demand " and allows you to watch, at any time, the indicated repetition of an event (Z) or challenge, as a simulation from various angles, allowing the user to freely control it in 3D.