RU2813497C1 - Sports timing based on camera system - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: method for determining the time of passage of an object through a timing line running across a sports track, including: receiving video frames, determining depth information, detecting one or more objects in the video frames using an object detection algorithm, determining at least one detected object in at least a portion of the video frames wherein at least one detected object passes through a timing line extending across the track, wherein the timing line is defined by a virtual plane located across the track at a predetermined distance from the camera system, and the determination of the object's passage is based on the coordinates of the virtual plane and depth information, and determining a transit time based on one or more times of one or more video frames containing at least one detected object passing the timing line.

EFFECT: increasing the accuracy of determining the time of passage of an object.

14 cl, 13 dwg

Description

Technical field

The present invention relates to camera system-based sports timing, and particularly, but not exclusively, to camera system-based sports timing methods and systems, as well as a computer program product enabling a computer system to perform such methods.

Prerequisites for creating the invention

Sports events such as car or motorcycle racing, cycling, athletics, drones and ice skating typically require accurate and fast time recording to track objects (people or vehicles) during the event. This timing system is typically based on an RFID (Radio Frequency IDentification) system, in which each event participant is provided with an RFID transponder, such as a UHF (Ultra High Frequency) backscatter tag or a Low Frequency (LF) tag. based on magnetic induction, which can be read by RFID readers located along the route. Such readers can be made in the form of an antenna mat, side antennas and/or antennas mounted on a frame above the track. Each transponder is configured to transmit packets on a specific frequency and insert a unique identifier into the packet so that the detector can associate the packet with a specific transmitter.

When a participant enters the reader's detection zone, the transponder begins to transmit signals that will be received by the reader. The signals may be time stamped so that an algorithm in the reader can determine the transit time based on one or more received transit signals. RFID technology can now be used to create highly reliable timekeeping systems for public events, with reliability of 99.8% or more. Similarly, RFID technology can be used to create highly accurate timing systems, allowing passage times to be determined with an accuracy of less than a hundredth of a second.

While providing reliable and accurate timing systems, the RFID system has certain disadvantages. One drawback is that each participant must have a transponder. Thus, before the marathon, each participant is provided with a bib number (BIB), which includes a UHF tag configured to convey an identifier (ID) uniquely associated with the BIB identifier. Additional disadvantages arise from the fact that UHF tags transmitting and receiving signals are quite sensitive to environmental influences, including reflection and absorption by objects and collisions (for example, when a large number of UHF tags simultaneously transmit signals that must be picked up by the reader). In addition, the operating frequency of the tags is close to the operating frequency of other mobile devices, which may cause unwanted interference effects.

In the prior art it has been proposed to use a timing system using a video system, i.e. a timing system based on the analysis of information received by one or more cameras that record a time series of video frames of competition participants. For example, US 6,433,817 describes a camera system for measuring a participant's progress through a virtual finish line. The camera includes a camera that is capable of capturing RGB (Red Green Blue) and infrared (IR) images. IR images are generated by a laser that transmits IR pulses to the finish line. In this way, the camera can detect depth information associated with objects, such as competitors passing the finish line. Likewise, DE 102006006667 presents a high-level system based on a camera system for timing a mass competition such as a marathon.

This document addresses the problem that in mass competitions a large number of participants cross the finish line at the same time or almost simultaneously. The paper suggests using multiple participant tokens so that the system can identify each participant. While this may seem like a reasonable solution in general, it does not describe the implementation of such a system. Implementing a video timing system that meets the reliability and accuracy requirements required for professional use is not a trivial task.

For example, Lynx technologies is one of the firms that currently sells a camera-based timing system. This system includes an (ultra-) high-speed photo finish timing camera positioned parallel to the finish line, combined with a video camera positioned in front of the finish line to identify various objects passing the finish line (almost) simultaneously. A photo finish camera is not suitable for mass events because too many participants cannot be visually identified. In addition, high-speed cameras are very expensive and therefore not suitable for easily determining lap times for mass events at multiple points along the course. Such a camera system cannot compete with the reliability, accuracy, and cost of RFID-based timing systems currently offered.

Thus, it follows from the above that there is a need in the art for improved timing of sports competitions using a video system that allows reliable timing and identification of participants in a sports competition, especially a sports competition with a large number of participants. In particular, there is a need in the art for video timing of sports events that is fast, reliable, easy to set up, and easy to use.

The essence of the invention

As will be appreciated by those skilled in the art, aspects of the present invention may be embodied in the form of a system, method, or computer program product.

Accordingly, aspects of the present invention may take the form of an all-hardware embodiment, an all-software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as "circuit", "module" or "system". The functions described in this document may be implemented as an algorithm executed by a computer microprocessor. In addition, aspects of the present invention may take the form of a computer program product embodied on one or more computer-readable media having computer-readable program code, for example, stored thereon.

Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable medium in the form of a signal or a computer-readable medium for storing data. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media may include the following: an electrical connection with one or more wires, a portable computer diskette, a hard disk, random access memory (RAM), read-only memory (ROM), EPROM (Erasable Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disk Readonly Memory), optical storage device, magnetic storage device or any a suitable combination of the above. As used herein, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with a system, hardware, or instruction execution device.

The computer-readable medium in the form of a signal may include a propagating data signal with computer-readable program code embodied therein, for example, in a baseband or as part of a carrier wave. Such a propagating signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium that is not a computer-readable data storage medium and that can transmit, distribute, or transport a program for use by or in connection with a system, instrument, or command execution device.

The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless communication, wired communication, optical fiber, cable, radio frequency, or the like, or any suitable combination thereof. The computer program code for performing the operations for aspects of the present invention may be written in any combination of one or more programming languages, including a functional or object-oriented programming language such as Java(TM), Scala, C++, Python, and the like. and conventional procedural programming languages, such as the C programming language or similar programming languages. The software code may run entirely on the user's computer, partially on the user's computer as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer, server, or virtualized server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local network (LAN, Local Area Network) or a wide area network (WAN, Wide Area Network), or the connection can be made to an external computer (for example, via the Internet using your internet service provider).

Aspects of the present invention are described below with reference to illustrations of block diagrams and/or block diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the invention. It should be understood that each block diagram and/or block diagram and combination of blocks in the block diagram illustrations and/or block diagrams may be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU, or GPU, Graphics Processing Unit), a general purpose computer, a special purpose computer, or other programmable data processing device, or other devices , which create means for implementing the functions/actions specified in the block or blocks of a block diagram and/or flowchart.

These computer program instructions may also be stored on a computer readable medium that may direct a computer, other programmable data processing device, or other devices to operate in a particular manner, such that the instructions stored on the computer readable medium create an article of manufacture including instructions that implement a function/action , specified in the block or blocks of the block diagram and/or block diagram.

Computer program instructions may also be loaded into a computer, other programmable data processing device, or other devices to cause a series of operating steps to be performed on the computer, other programmable device, or other devices to create a computer-implemented process such that the instructions that are executed on the computer or other programmable device, provide processes for implementing the functions/actions specified in the block or blocks of the block diagram and/or block diagram.

The block diagrams and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products in accordance with various embodiments of the present invention. In this regard, each block in a block diagram or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions to implement specified logical function(s). It should also be noted that in some alternative implementations, the functions shown in the blocks may not be executed in the order shown in the figures. For example, two blocks shown sequentially may actually be executed substantially simultaneously, or sometimes the blocks may be executed in reverse order, depending on the functionality involved. It should also be noted that each block of the block diagram and/or block diagram illustrations and combinations of blocks in the block diagram and/or block diagram illustrations may be implemented by special purpose hardware systems that perform the mentioned functions or actions, or combinations of special purpose hardware and computer commands

The purpose of the embodiments in this invention is to reduce or eliminate at least one of the disadvantages known in the prior art. In one aspect, the invention may relate to a method for determining the time of passage of an object through a timing line extending across a sports track, comprising: receiving a sequence of video frames captured by at least one camera system, preferably a 3D camera system; wherein each video frame represents an image of a scene with one or more objects, such as a person, animal or vehicle, moving along the track, and each video frame has a timestamp; determining depth information, such as depth maps, for a sequence of video frames, wherein the depth information includes information regarding the distance between one or more objects in an image of the video frame and the camera system; detecting one or more objects in the video frames using an object detection algorithm, wherein said objects detected by the detection algorithm are determined by the one or more detected objects; determining on video frames a detected object crossing a timing line running across the sports track, wherein the timing line is determined by a virtual plane located across the track at a given distance from the camera system, the passage is determined based on the coordinates of the virtual plane and depth information; and determining a transit time based on a timestamp of the video frame containing the detected object crossing the timing line.

In an embodiment of the invention, the method may further include applying a feature analysis algorithm to one or more detected objects in the video frames, the feature analysis algorithm determining identifying features for the one or more detected objects in the video frames; and determining the identity of the detected object for which the transit time is determined based on identification features of the detected object that has passed through the timing line.

In an embodiment of the invention, identifying features of a detected object include one or more optically readable identification markers, such as a bib number or a printed mark; and/or one or more characteristics of the shape and/or color of the detected object; and/or, if the detected object is an animal or human, one or more biometric identifiers of the detected object.

In an embodiment of the invention, the object detection algorithm and the feature analysis algorithm may be part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, that learns to detect one or more objects in a video frame and determine identifying features associated with the detected objects.

In an embodiment of the invention, detecting one or more objects in video frames may include: determining one or more regions of interest (ROI, Region Of Interest) in the video frame, each ROI containing pixels representing an object; determining identification features in one or more areas of the ROI; and identifying an object in the ROI based on certain identification features.

In an embodiment of the invention, the camera system may comprise a plurality of camera modules, preferably two camera modules, forming a stereo camera, wherein the stereo camera is configured to generate at least a first video frame and a second video frame of the scene at each instant, and a depth map is determined based on a disparity mapping algorithm , configured to determine the disparity between the pixels of the first and second video frame.

In an embodiment of the invention, the transit time may be determined based on a video frame of the scene in which a predetermined portion of the detected object has passed the virtual plane.

In another aspect, the invention may relate to a method for determining the time of passage of objects through a timing line extending across a sports track, comprising:

receiving video frames from a plurality of camera systems, preferably time-synchronized camera systems, the plurality of camera systems capturing a sports track scene from different viewing angles, the video frames being images of the scene containing one or more objects, such as a person, animal or vehicle, moving along the track, each of the video frames having a time stamp; determining depth information, such as depth maps, based on the received video frames, wherein the depth information includes information regarding the distance between one or more objects in an image of the video frames and the plurality of camera systems; detecting one or more objects in the video frames using an object detection algorithm, wherein the one or more objects detected by the detection algorithm determines the one or more detected objects; determining a detected object of video frames passing through a timing line extending across the sports track, the timing line being defined by a virtual plane located across the track at specified distances from a plurality of camera systems, the detection of the passage being made based on the coordinates of the virtual plane and depth information maps; and determining a transit time based on one or more timestamps of one or more video frames containing the detected object passing through the timing line.

In an embodiment of the invention, the method may further include: applying a feature analysis algorithm to one or more detected objects in the video frames, wherein the feature analysis algorithm determines identifying features for the one or more detected objects in the video frames; and determining the identity of the detected object for which the transit time is determined based on identification features of the detected object that has passed through the timing line.

In a further aspect, the invention may relate to a method for calibrating a timing system configured to determine the time of passage of an object through a timing line extending across a sports track, the method comprising: receiving a sequence of video frames captured by a camera system, preferably a 3D camera system, of the timing system, each video frame representing an image of a scene including a track and calibration markers located on opposite sides of the track; determining one or more depth maps based on the video frames, wherein the depth map contains information about the distance between one or more objects in an image of the video frame; using one or more depth maps to determine the distance between the calibration markers and the camera system; determining the coordinates of a virtual plane located across the track between the markers, the virtual plane defining a timing line for the timing system; and storing the coordinates of the virtual plane in the memory of the timing system.

In another aspect, the invention relates to a system for determining the time of passage of an object crossing a timing line extending across a sports track, the system may comprise: at least one camera system connected to a computer; a computer comprising a computer-readable storage medium having computer-readable program code embodied thereon, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein in response to execution of the computer-readable program code, the processor is configured to perform executable operations including: receiving a sequence of video frames captured at least one camera system, preferably a 3D camera system, wherein each video frame is an image of a scene with one or more objects, such as a person, animal or vehicle, moving along the track, and each video frame has a timestamp; determining depth maps for a sequence of video frames, wherein the depth map contains information regarding the distance between one or more objects in an image of the video frame and the camera system; detecting one or more objects in the video frames using an object detection algorithm, wherein the one or more objects detected by the detection algorithm determines the one or more detected objects; determining a detected object in one of the video frames that passes through a timing line extending across the athletic track, the timing line being defined by a virtual plane located across the track at a specified distance from the camera system, determining the passage based on the coordinates of the virtual plane and depth maps, the coordinates being preferred stored on a computer readable storage medium; and determining a transit time based on a timestamp of a video frame containing the detected object passing through the timing line.

In an embodiment of the invention, the operations performed may further include: applying a feature analysis algorithm to one or more detected objects in the video frames, wherein the feature analysis algorithm determines identifying features for the one or more detected objects in the video frames; and determining the identity of the detected object for which the transit time is determined based on identification features of the detected object that has passed the timing line.

In yet another aspect, the invention may relate to a calibration module for a timing system configured to determine the time of passage of an object through a timing line extending across a track, the module comprising: receiving a sequence of video frames captured by a camera system, preferably a 3D camera system, the timing system, where each video frame is an image of a scene including a track and calibration markers located on opposite sides of the track; determining one or more depth maps based on the video frames, wherein the depth map contains information regarding the distance between one or more objects in an image of the video frame; using one or more depth maps to determine the distance between the calibration markers and the camera system; determining the coordinates of a virtual plane located across the track between the markers, wherein the virtual plane defines a timing line for the timing system; and storing the coordinates of the virtual plane in the memory of the timing system.

The invention may also relate to a computer program or set of computer programs containing at least one piece of software code, or to a computer program product storing at least one piece of software code, wherein the portion of the software code is configured when run on a computer system to perform any of the method steps described above.

The invention may further relate to a computer-readable storage medium storing at least one portion of software code, wherein the portion of the software code, when executed or processed by a computer, is configured to perform any of the steps of the method as described above.

The invention will now be illustrated with reference to the accompanying drawings, which schematically show embodiments according to the invention. It should be understood that the invention is in no way limited to these specific embodiments.

Brief description of drawings

FIG. 1 is a diagram of a timing system according to an embodiment of the invention.

FIG. 2 illustrates a camera system for use in a timing system according to an embodiment of the invention.

FIG. 3 shows a disparity display of images produced by the camera system.

FIGS. 4A and 4B depict calibration of a timing system according to an embodiment of the invention.

FIG. 5 is a block diagram of a method for calibrating a timing system according to an embodiment of the invention.

FIGS. 6A and 6B illustrate a method for determining transit time according to an embodiment of the invention.

FIG. 7 illustrates a method for identifying a synchronized object according to an embodiment of the invention.

FIG. 8 is a block diagram of a method for determining transit time according to an embodiment of the invention.

FIG. 9 illustrates a system for determining transit time according to an embodiment of the invention.

FIG. 10 depicts some illustrative images of objects passing through a virtual plane of a timing system, according to an embodiment of the invention.

Fig. 11 is a block diagram illustrating an example of a data processing system that can be used to implement the methods and software products described in this application.

Detailed description

FIG. 1 illustrates a timing system according to an embodiment of the invention. In particular, the figure depicts a video-based sports timing system 100 including one or more camera systems 102 _1,2 controlled by a computer 104 _1,2 . Each camera system may be configured to capture a scene of the athletic track 106 and determine depth information associated with the captured scene. For example, in an embodiment of the invention, the depth information may include so-called depth maps for video frames generated by the camera system's image sensor. A depth map of a video frame, such as an RGB video frame, can be represented as a pixel image containing pixel values representing a distance value for each pixel in the video frame. The distance value can determine the distance between the camera (camera image plane) and objects in video frames.

For example, a group of pixels in a video frame may be part of an object in the scene that is being imaged by the camera system. In this case, the depth map can indicate the relative distance between the camera (the shooting point) and the surface of an object in the scene. Therefore, while capturing a sequence of video frames with timestamps of an object, such as an athlete or a vehicle moving on a sports track, corresponding depth maps can provide information about the distance between the moving object in the video frames and the (static) camera system as a function of time.

Camera systems are known that are capable of generating a depth map. For example, in an embodiment of the invention, the camera may be implemented as a 3D camera system, such as a stereo camera, comprising two or more camera modules, each camera module having its own lens system. An example of a top view of such a three-dimensional (3D) imaging system is shown in FIG. 2. As shown in this figure, a stereo camera 202 may be positioned along a side of the track 204. The stereo camera has (at least) two camera modules 204 _1,2 with a parallel optical axis 206 _1,2 for observing a treadmill scene including objects, such as calibration markers 210 _1,2 and/or athletes (as shown in Fig. 1). This captures two images of the same scene from different viewpoints. Such a stereo camera can be used to determine depth maps and/or 3D images. A 3D camera system can have more than two camera modules so that multiple images can be used to calculate the depth map. In this way, the accuracy of depth maps can be improved.

The 3D camera system (or a computer system controlling the camera system) may include a module for calculating a depth map based on video frames captured by two (or more) camera modules. In an embodiment of the invention, the module may use disparity mapping techniques to calculate a depth map based on images generated by two image sensors. Figure 3 schematically shows the principle of disparity display. The 3D camera system contains two camera modules separated by a distance called base 308. The two camera modules can be synchronized such that two video frames 302 _1,2 of the same object from different viewpoints are generated at any given time.

To compute a depth map based on these video frames, a matching algorithm can be performed to match the corresponding pixels of the left and right video frame. Therefore, the object 300 displayed by the two synchronized camera modules is located at the same position 304 _1,2 but separated by a base distance 308. In this case, the object will appear in the same positions in both images. The distance between objects in the left and right images is known as disparity 306. An algorithm for constructing a disparity map from two images is known as a stereo matching algorithm. There are various stereo image matching algorithms that need to be accurate and fast for real-time applications.

The 3D camera system that is used in the embodiments of this application is not limited to stereo-based imaging techniques, and other 3D imaging techniques may also be used. For example, the depth map can be generated based on the RGBAR technique (used by Kinect) or three-dimensional (3D) time-of-flight (TOF) technique, or a combination of both. Additionally, to increase the viewing angle of the camera system, some embodiments of the invention may use one or more wide-angle camera systems, such as a 180-degree camera or a 360-degree camera. Depth maps can also be created for types of video formats such as 360-degree video or immersive video environments, generated using special 360-degree camera systems in which video is projected onto a 2D video frame using, for example, equidistant projection.

As shown in FIG. 1, one or more 3D camera systems 102 _1,2 may be located on the side or above the athletic track. Additionally, the camera systems may be aligned and calibrated such that each of the 3D camera systems captures the same track scene 114, including objects moving along the track. For this purpose, one or more calibration markers 110 _1,2 can be used. One or more calibration markers may be located on one side of both sides, for example, on opposite sides of the track. These calibration markers can be used by the timing system to calculate the coordinates of a virtual plane 112 located across the path between the two calibration markers. After calibration, the relative distance between the 3D camera system and each position on the virtual plane is known. This allows the timing system to determine, based on time-stamped video frames and associated depth maps, at what point in time a moving object passed the virtual plane.

The 3D camera system calibration process is required to accurately determine the position and orientation of the calibration markers under any environmental conditions. Thus, the calibration markers are designed to have a predetermined shape and/or color combination, and so that during calibration, the object detection program can easily and accurately determine the position (edges) of the markers in the video frames so that the coordinates of the virtual plane can be accurately determined. When the (calibrated) timing system is running, the 3D camera system can capture video footage of athletes passing through the virtual plane. Although the figure shows a camera system along the side of a treadmill, in other embodiments of the invention one or more camera systems may be installed above the track using a suitable mounting structure.

As will be described in more detail below, the timing system depicted in FIG. 1 is configured to determine the time of passage of objects through a virtual plane. In addition, the timing system is configured to identify objects for which transit times are determined. Identification may be based on identification markers associated with objects. For example, in the case of athletes, identification markers may include, but are not limited to, a bib number, colors, biometric information, and the like. In the case of vehicle identification, the markers may include characteristics of the vehicle, such as color, shape, brand names, etc.

The computer for controlling one or more 3D camera systems and performing the calibration and timing methods may be implemented as a stand-alone computer or a set of wirelessly connected computers. For example, 3D camera systems that are used to determine travel times based on a virtual plane located transverse to the track may be controlled by a computer that includes a wireless interface for wireless communication with computers that control other 3D camera systems.

Multiple timing systems, as shown in FIG. 1, may be located at different locations along the route. Thus, each timing system can determine transit times based on video frames of objects passing through the virtual plane. Timing systems can support a wireless protocol that allows a mesh network of timing systems to be established along the course. Thus, information determined by one timing system can be used by one or more other timing systems that are part of the mesh network.

The data processing methods used by the timing system to calculate depth maps and analyze video frames may require real-time image processing, so in some embodiments of the invention, a special purpose processor, such as a graphics processing unit (GPU), may be used to perform portions of the calibration process and synchronization that requires large computing resources. In other embodiments, one or more 3D camera systems may be connected to cloud resources that can perform computationally intensive portions of the processes. The CPU clock or Global Positioning System (GPS) clock signal can be used to associate video frames with time information. For example, in an embodiment of the invention, all or at least a portion of the video frames may be time-locked by time-stamping the video frames.

The timing system in FIG. 1 can be operated using a single 3D camera. Alternatively, two or more 3D camera systems from different viewing angles can be used to capture video frames of a scene. In this case, the 3D camera systems can be time-synchronized so that time-stamped video frames from different 3D camera systems can be easily combined and analyzed. Numerous 3D camera systems can be used to handle collisions, e.g. events when several objects pass through the virtual plane.

Fig. 4A and 4B depict an embodiment of a timing system that is being calibrated and a timing system that is operating, respectively. As shown in FIG. 4A, a camera system 402 is located along a side of the track 406. At a given distance 406, one or more calibration markers 410 _1,2 may be located on one or both sides of the track. The camera's field of view is directed toward the track so that it captures the scene including one or more calibration markers. The calibration program is then executed by the camera system computer 404. During calibration, the camera generates video frames, and depth maps of a scene including one or more calibration markers are determined based on the generated video frames (eg, as described with reference to FIGS. 2 and 3). An object detection program can be used to detect calibration markers in video frames (such as RGB images).

One or more calibration markers may be designed to have design features that provide accurate calibration under a variety of environmental conditions.

For example, the shape, edges and/or colors of the marker can be designed to enable accurate detection in images. The depth map associated with the video frames can be used to determine the distance between the camera and the detected calibration markers. Alternatively, if a sufficiently accurate depth map can be constructed, the object detection program can also determine the position of the calibration markers directly from the depth map.

Once the position of one or more markers is detected, the computer can determine a virtual plane located between the two calibration markers. The virtual plane can be used as the location at which the timing system determines the time of passage. This virtual plane may be located within a rectangular three-dimensional (3D) volume 412 in space, where the width of the volume may be determined by gauge markers and the height and depth of the volume may be determined by a computer. The 3D volume may define a 3D detection zone in which the timing system will receive video data (eg, video frames) to determine the time of passage and to identify an object associated with the time of passage.

The same calibration process can be used to install and calibrate one or more additional 3D camera systems along a track so that each of these 3D camera systems can capture video frames of objects passing through the same 3D detection zone from a different viewing angle. Camera systems can (wirelessly) communicate with each other so that the video capture process can be time-synchronized. Thus, at one point in time, each of the camera systems will process one or more time-stamped video frames of the sports track that include a 3D detection zone taken from a specific viewing angle. Time-stamped video frames (and associated depth maps) for different viewing angles can be used to determine the transit times of objects passing through the virtual plane and to identify objects for which transit times have been determined.

FIG. 5 is a block diagram of a method for calibrating a timing system according to an embodiment of the invention. This method can be used to calibrate a timing system containing calibration markers, as described with reference to FIG. 1 and FIG. 4A. As shown in the block diagram, the method may begin by pointing the timing system's three-dimensional camera system at one or more calibration markers located on one or both sides of the track. One or more calibration markers may indicate the position of a virtual plane located across the track (step 502). For example, in the case of two calibration markers on opposite sides of the track, the virtual plane may be located between the two markers. In a next step 504, a sequence of video frames may be captured by a 3D camera system, where each video frame may represent an image of a scene including a track and calibration markers.

Thereafter, depth information, such as one or more depth maps, can be determined based on the video frames; the depth map may include information regarding the distance between one or more objects in the image and the 3D camera system (step 506). For example, a depth map can be generated based on two video frames generated by two camera modules in a stereo camera, and the disparity mapping can be used to generate a depth map in the same manner as described with reference to FIGS. 2 and 3 above. The depth information may include a depth map containing, for each video frame (or at least a substantial portion thereof), information regarding the distance between objects in the captured video frames and the 3D camera. The depth information may be used to determine the distance between each of the calibration markers and the 3D camera (step 508). The coordinates of a virtual plane located across the path between the calibration markers can then be calculated. The virtual plane can be used by a timing system to determine the time at which objects pass through the virtual plane. In addition, the coordinates of a 3D volume oriented across the sports track between the calibration markers are calculated. The 3D volume spans the virtual plane and defines a detection volume to determine the transit time and identify an object associated with the transit time (step 510). The coordinates of the virtual plane and the associated 3D volume of the detection zone may be stored in the memory of the timing system (step 512).

As shown in FIG. 4B, the timing system no longer requires calibration markers. A calibrated camera system will use a virtual plane in the 3D detection zone to determine the transit time and one or more identifying features of an object passing through the virtual plane. This process will be referred to as a pass event, which will be described in more detail with reference to FIGS. 6 and 7.

Fig. 6A and 6B depict a method for determining transit time according to an embodiment of the invention. FIG. 6A shows three photographs of a moving object (in this case, an athlete) passing through the 3D detection zone of a timing system, as described with reference to embodiments of this application. The 3D detection zone can be set using a calibration method as described above with reference to FIG. 5. As shown in FIG. 6B, the 3D detection zone 612 includes a virtual plane 614 located across the track, with the normal to the virtual plane being substantially parallel to the direction of the track. The virtual plane divides the 3D detection zone into a first part 616 ₁ in which an object approaches and traverses the virtual plane, and a second part 616 ₂ in which a moving object traverses and moves away from the virtual plane.

Thus, when an object moves along the track, the 3D camera system will capture images (pairs of images in the case of a stereo camera) of the scene including the 3D detection zone. For each image (video frame), the three-dimensional (3D) image processing system can calculate a depth map. An object detection and tracking algorithm can be used to detect and track a given object, such as a human object, or an object representing an object, in subsequent video frames. Algorithms for detecting objects are known. Based on the depth maps, the computer can determine that the detected object is included in the first part of the three-dimensional detection zone. In this case, the computer can begin storing video frames and their associated depth maps in a buffer until the object leaves the 3D detection zone through the second part. In another embodiment of the invention, only pairs of video frames are stored and depth maps are determined later. These video frames and depth maps are used by the computer to determine the transit time and identify the object associated with the transit time.

Fig. 6A and 6B illustrate a method for determining transit time according to an embodiment of the invention. In particular, FIG. 6A depicts an object (in this example, an athlete) moving through the 3D detection zone of a 3D camera system. 6A depicts three sample sequences 602 _1-3 of time-stamped video frames as the athlete moves through the 3D detection zone. These video frames are captured by a 3D camera system and stored in a computer. As shown in FIG. 6B, the first video frame 602 ₁ is captured at time T1, that is, while most of the athlete's body is still in the first portion 602 ₁ of the 3D detection zone. At time T2, a second video frame 602 d is captured as the athlete moves on and passes the virtual plane 614 of the 3D detection zone. Finally, at time T3, a third video frame 602 ₃ is captured in which the athlete has moved into the second 3D detection zone portion 616 ₂ .

The passage time module in the timing system's computer can analyze a sequence of time-stamped video frames to determine at what point in time the athlete passed the virtual plane. For this purpose, an object detection and classification algorithm can be applied to each video frame. To this end, the algorithm may determine regions of interest of the video frame 608 _1-3 (ROIs) that belong to the object. Additionally, for each of these ROIs, the algorithm can classify the pixels as belonging to the athlete or not (belonging to the background). In addition, the depth map associated with each of the video frames can be used to determine the distance values belonging to those pixels that are classified as belonging to the object. These distance values can be compared to the distance between the camera and the virtual plane. Thus, when the 3D camera system captures an object intersecting the virtual plane, for each video frame the portion of the object's pixels intersecting the virtual plane can be determined. This can be seen in the video frames of Fig. 6A, where the gray areas define the pixels (parts) of the object that crossed the virtual plane.

For the video frame at time T1, only pixels 604 ₁ representing part of the hand and pixels 606 ₁ representing the athlete's shoe are associated with distance values less than the distance between the virtual plane and the 3D camera system. Likewise, for a video frame at time T2, pixels 604 ₁ representing part of the upper body and pixels 606 ₂ representing part of the leg are associated with distance values less than the distance between the virtual plane and the 3D camera system. Finally, for the video frame at time T3, all pixels 608 representing the athlete are associated with distance values less than the distance between the virtual plane and the 3D camera system. Based on this analysis, the computer can determine that at T2 a significant portion of the athlete's body has crossed the virtual plane. For example, the computer may determine that if the portion of an object that has crossed the virtual plane is greater than a certain threshold value, then the athlete is determined to have crossed the plane. Therefore, timestamp T2 may in this case determine the time of passage 610, in this example 2:34. Various rules can be defined to determine whether an object has crossed the virtual plane.

FIG. 7 illustrates a method for identifying a timed object according to an embodiment of the invention. Once the transit time of an object is established, the identity of the object can be established. For this purpose, the stored video frames can be analyzed by the object identification module. Regions of interest (ROIs) in video frames can be subjected to a feature analysis process that looks for predefined features in the ROI. For example, as shown in FIG. 7, the ROI 703 of the T3 video frame 702 of FIG. 6 may be subjected to processing by a feature analysis algorithm, which may be configured to identify predetermined features, such as predetermined markers, for example, bib number 704 ₁ , or code such as the 704 g quick response code (QR, Quick Response). Any unique identifying feature may be used, including shoe type 704 ₃ , biometric markers, colored or specially shaped markers, or a combination thereof. The analysis can be performed based on a set of ROIs from different video frames during the 3D detection zone. It is often the case that the video frame used to determine transit time is not suitable for identification purposes. In addition, the object identification module may determine that a time-stamped video frame (eg, a video frame at T2 in the example of FIG. 6) that represents an image of an object crossing the virtual plane is not suitable for reliable object identification. In this case, one or more other video frames (eg, the video frame in T3 in the example of FIG. 6) may be determined to be suitable for reliably identifying the timed object. Any known object detection algorithm can be used to identify features in a region of interest (ROI), including conventional computer vision algorithms, machine learning and deep learning technologies. For example, in one embodiment of the invention, one or more deep neural networks may be used to determine an ROI in a video frame that contains an object and to determine certain features associated with the object.

FIG. 8 is a block diagram of a method for determining transit time according to an embodiment of the invention. As shown in this figure, the method may begin by obtaining a sequence of video frames captured by at least one camera system, preferably a 3D camera system, wherein each video frame represents an image of one or more objects, such as a person, animal or vehicle, moving along a track, and each video frame has a timestamp (step 802). Thereafter, depth information, such as one or more depth maps, may be determined for the sequence of video frames, wherein the depth information may include information regarding the distance between one or more objects in the video frame image and the camera system (step 804). Typically, a 3D camera system may be a stereo camera such that it produces pairs of video frames that can be used to determine a depth map. Thereafter, one or more objects in the video frames may be detected using an object detection algorithm, wherein the one or more objects detected by the detection algorithm determines the one or more objects detected (step 806). Thus, in each video frame an object can be identified and the distance between the identified object and the 3D camera system can be determined.

In an embodiment of the invention, the object detection step may include identifying regions of interest (ROIs) containing the object, and for each ROI, processing the pixels in the ROI using a classification algorithm to classify whether the pixel represents part of the object or part of the background.

In addition, a detected object in video frames may be determined to pass through a timing line extending across the athletic track, wherein the timing line is defined by a virtual plane located across the track at a specified distance from the camera system, wherein the determination of the passage is based on virtual plane coordinates and depth maps (step 808). Therefore, the distance between the 3D camera system and the virtual plane can be compared with the distance between the 3D camera system and the detected object. The transit time is then determined based on a time point, such as a timestamp, associated with one or more video frames containing the detected object passing through the timing line (step 810). For example, for this purpose, one or more video frames can be defined in which part of the object that has passed through the virtual plane has certain dimensions.

Therefore, certain rules can be used to determine whether an object has passed through the virtual plane. A point in time, such as a timestamp, associated with a video frame that shows this situation determines the transit time. The feature analysis algorithm may then be applied to the one or more detected objects in the video frames, wherein the feature analysis algorithm determines identifying features for the one or more detected objects in the video frames (step 812) and the identity of the detected object, for which a travel time can be determined at based on the identification characteristics of a detected object that passed through the timing line.

In an embodiment of the invention, the object detection algorithm and the feature analysis algorithm are part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, that learns to detect one or more objects in a video frame and determine identifying features associated with the detected objects.

Thus, different images from a sequence of video frames can be used to identify the object that crossed the virtual plane at the moment of its passage. Therefore, the video frame used to determine the transit time of an object may be different from the one or more video frames that are used to determine the identity of the object.

FIG. 9 illustrates a system for determining transit time according to an embodiment of the invention. As shown in the figure, the system includes one or more camera systems 901 _1,2 connected to a computer system that is configured to control the camera systems and process video frames 903 _1,2 generated by the camera systems. Each of the camera systems is calibrated in accordance with the calibration method described in this application so that it captures a track scene including moving objects that pass through a timing line represented by virtual plane 902. The computer system may include an event analyzer 904 to perform object detection and analyzing features of objects in video frames using a suitable algorithm. Additionally, the event analyzer can generate depth information, such as depth maps associated with video frames. The transit module 906 can determine the transit time based on depth information and virtual plane coordinates. In addition, the object identification module 908 may identify objects that are detected and associated with the transit time by the transit time module. The transit time values and the identifiers (IDs) of the object that has passed may be stored in the computer's memory. In addition, an image of the passage of the timed object can be stored. This information may be transmitted via (wireless) interface 912 to network server 914.

FIG. 10 shows some illustrative images of objects passing through a timing line of a timing system according to an embodiment of the invention. As shown in these images, the calibration tools are used to set a virtual plane between the markers. The plane is positioned perpendicular to the track to allow objects such as athletes to cross it. The images also show that the timing system is able to detect, as an object passes, the part of the object that has passed the virtual plane (indicated by the white parts) and the part of the object that has not passed the finish line.

Fig. 11 is a block diagram illustrating an example of the data processing systems described herein. The data processing system 1100 may include at least one processor 1102 coupled to the memory elements 1104 via a system bus 1106. Thus, the data processing system may store program code in the memory elements 1104. In addition, processor 1102 may execute program code obtained from memory elements 1104 via system bus 1106. In one aspect, the data processing system may be implemented as a computer suitable for storing and/or executing program code. However, it should be understood that the data processing system 2100 may be implemented as any system comprising a processor and memory that is capable of performing the functions described in this specification.

Memory elements 1104 may include one or more physical memory devices, such as, for example, local memory 1108 and one or more mass storage devices 1110. Local memory may refer to RAM or other non-persistent memory device(s) commonly used during the actual execution of program code. The storage device may be implemented as a hard disk or other permanent storage device. Processing system 1100 may also include one or more caches (not shown) that provide temporary storage of at least some program code to reduce the number of times program code must be retrieved from storage device 1110 during execution.

Input/output devices (I/O, Input/Output), depicted as input device 1112 and output device 1114, can optionally be connected to the data processing system. Examples of the input device may include, but are not limited to, a keyboard, a pointing device such as a mouse, and the like. Examples of output devices may include, but are not limited to, a monitor or display, speakers, and the like. An input device and/or an output device may be coupled to the data processing system either directly or through intermediate input/output (I/O) controllers. The network adapter 1116 may also be coupled to the data processing system to allow it to communicate with other systems, computer systems, remote network devices, and/or remote storage devices via intermediate private or public networks. The network adapter may include a data receiver for receiving data that is transmitted by said systems, devices and/or networks for said data, and a data transmitter for transmitting data to said systems, devices and/or networks. Modems, cable modems, and Ethernet cards are examples of various types of network adapters that can be used with the data processing system 1100.

As shown in FIG. 11, memory elements 1104 may store application 1118. It should be understood that processing system 1100 may further implement an operating system (not shown) that may facilitate execution of the application. The application, implemented as executable program code, may be executed by a data processing system 1100, such as a processor 1102. In response to execution of the application, the data processing system may be configured to perform one or more operations, which are described in detail herein.

In one aspect, for example, the data processing system 1100 may represent a client data processing system. In this case, application 1118 may be a client application that, when executed, configures data processing system 2100 to perform various functions described herein with reference to a “client.” Examples of a client may include, but are not limited to, a personal computer, laptop computer, mobile phone, or the like.

In another aspect, the data processing system may be a server. For example, the data processing system may be a server (Hypertext Transfer Protocol (HTTP) request processing), in which case application 1118 may, when executed, configure the data processing system to perform server (HTTP) operations. In another aspect, the data processing system may be a module, block, or function, as defined herein.

The terminology used herein is intended to describe specific embodiments only and is not intended to limit the invention. The singular forms used herein are intended to also include plural forms unless the context clearly indicates otherwise. In addition, it should be understood that the terms “comprises” and/or “comprising” when used herein define the presence of the claimed features, integers, steps, operations, elements and/or components, but do not exclude the presence or addition of one or more other features , whole, stages, operations, elements, components and/or their groups.

The respective structures, materials, acts and equivalents of all means or steps plus functional elements in the claims below are intended to include any structure, material or act to perform a function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limiting of the invention in its disclosed form. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. An embodiment of the invention has been selected and described to best explain the principles of the invention and its practical application and to enable others skilled in the art to understand the invention for various embodiments with various modifications as are suitable for the particular intended use.

Claims (53)

1. A method for determining the time of passage of an object through a timing line running across a sports track, including: receiving video frames, preferably a sequence of video frames, captured by at least one camera system, preferably a 3D camera system, wherein each video frame represents an image of a scene with one or more objects, such as a person, animal or vehicle, moving along a track, and each video frame is associated with a point in time, for example, has a timestamp; determining depth information, such as one or more depth maps, for at least a portion of the video frames, wherein the depth information includes information regarding a distance between at least one of the one or more objects in an image of the video frame and the camera system; detecting one or more objects in the video frames using an object detection algorithm, wherein the one or more objects detected by the detection algorithm determines the one or more detected objects; determining at least one detected object in at least a portion of the video frames, wherein the at least one detected object passes through a timing line extending across the track, wherein the timing line is defined by a virtual plane located across the track at a predetermined distance from the camera system, and determining the passage of the object is based on virtual plane coordinates and depth information; And determining a transit time based on one or more times of one or more video frames containing at least one detected object passing the timing line. 2. The method according to claim 1, additionally including: applying the feature analysis algorithm to the one or more detected objects in the video frames, the feature analysis algorithm determining identifying features for the one or more detected objects in the video frames; And determining the identity of the detected object, for which the transit time is determined, based on the identification characteristics of the detected object that has passed the timing line. 3. The method of claim 2, wherein the identification features of the detected object include one or more optically readable identification markers, such as a bib number or a printed mark; and/or one or more characteristics of the shape and/or color of the detected object; and/or, if the detected object is an animal or human, one or more biometric identifiers of the detected object. 4. The method of claim 2 or 3, wherein the object detection algorithm and the feature analysis algorithm are part of a machine learning algorithm, preferably a deep learning algorithm, such as a convolutional deep neural network system, which learns to detect one or more objects in a video frame and determine the identification signs associated with detected objects. 5. Method according to any one of paragraphs. 1-4, wherein detecting one or more objects in video frames includes: defining one or more regions of interest (ROIs) in the video frame, each ROI containing pixels representing an object; determining identification features in one or more areas of the ROI; And identifying an object in the ROI based on identification features that are defined. 6. Method according to any one of paragraphs. 1-5, in which the camera system includes a plurality of camera modules, preferably two camera modules, forming a stereo camera, wherein the stereo camera is configured to generate at each time at least a first video frame and a second video frame of the scene, and a depth map is determined based on the mapping algorithm disparity configured to determine the disparity between pixels of the first and second video frame. 7. Method according to any one of paragraphs. 1-6, in which the travel time is determined based on a video frame of the scene in which a given part of the detected object has passed the virtual plane. 8. A method for determining the time of passage of objects through a timing line running across a sports track, including: receiving video frames from a plurality of camera systems, preferably time synchronized, wherein the plurality of camera systems capture a sports track scene from different viewing angles, the video frames being images of the scene containing one or more objects, for example a person, animal or vehicle, moving along the track, each of the video frames being associated with a point in time, for example, having a time stamp; determining depth information, for example, one or more depth maps, based on the received video frames, wherein the depth information includes information regarding the distance between one or more objects in an image of the video frame and at least one of the plurality of camera systems; detecting one or more objects in the video frames using an object detection algorithm, wherein the one or more objects detected by the detection algorithm determines the one or more detected objects; determining at least one detected object in at least a portion of the video frames, wherein the at least one detected object passes through a timing line extending across the track, wherein the timing line is defined by a virtual plane located across the track at predetermined distances from the plurality of camera systems , and determining the passage of an object is based on virtual plane coordinates and depth information; And determining a transit time based on one or more times of one or more video frames containing at least one detected object passing through the timing line. 9. The method according to claim 8, additionally including: applying a feature analysis algorithm to one or more detected objects in video frames, wherein the analysis algorithm features defines identifying features for one or more detected objects in video frames; And determining the identity of the detected object, for which the transit time is determined, based on the identification characteristics of the detected object that has passed the timing line. 10. A method for calibrating a timing system configured to determine the time of passage of an object through a timing line extending across a track, the method comprising: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, of said timing system, wherein each video frame is an image of a scene including a track and one or more calibration markers, preferably at least two calibration markers located in opposite directions sides of the path; determining depth information, such as one or more depth maps, based on the video frames, wherein the depth information includes information regarding a distance between one or more objects in an image of the video frame; using the depth information to determine a distance between the at least one calibration marker and the camera system; determining the coordinates of a virtual plane located across the track at the location of one or more calibration markers, preferably between at least two calibration markers located on opposite sides of the track, wherein the virtual plane defines a timing line for the timing system; And saving the coordinates of the virtual plane in the memory of the timing system. 11. A system for determining the time of passage of an object through a timing line running across a sports track, containing: at least one camera system connected to a computer; a computer comprising a computer-readable storage medium having computer-readable program code embodied thereon, and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein, in response to execution of the machine-readable program code, the processor is configured to perform executable operations including: receiving video frames, preferably a sequence of video frames, captured by at least one camera system, preferably a 3D camera system, wherein each video frame is an image of a scene with one or more objects, such as a person, animal or vehicle, moving along a track, and each video frame associated with a point in time, for example, has a timestamp; determining depth information, such as one or more depth maps, for at least a portion of the video frames, wherein the depth information includes information regarding a distance between at least one of the one or more objects in an image of the video frame and the camera system; detecting one or more objects in the video frames using an object detection algorithm, wherein the one or more objects detected using the detection algorithm determines the one or more detected objects; determining at least one detected object in at least a portion of the video frames, wherein the at least one detected object passes through a timing line extending across the track, wherein the timing line is defined by a virtual plane located across the track at a predetermined distance from the camera system, and the passage determination is based on virtual plane coordinates and depth information; And determining a transit time based on one or more times of one or more video frames containing at least one detected object passing through the timing line. 12. The system according to claim 11, in which the operations performed additionally include: applying the feature analysis algorithm to the one or more detected objects in the video frames, the feature analysis algorithm determining identifying features for the one or more detected objects in the video frames; And determining the identity of a detected object, for which the passage time is determined, based on the identification characteristics of the detected object that has passed the timing line. 13. A calibration module for a timing system configured to determine the time of passage of an object through a timing line extending across a track, the module comprising: receiving video frames, preferably a sequence of video frames, captured by a camera system, preferably a 3D camera system, a timing system, each video frame representing an image of a scene including a track and one or more calibration markers, preferably at least two calibration markers located on opposite sides of the track; determining depth information, such as one or more depth maps, based on the video frames, wherein the depth information includes information regarding a distance between one or more objects in an image of the video frame; using the depth information to determine a distance between the at least one calibration marker and the camera system; determining the coordinates of a virtual plane located across the track at the location of one or more calibration markers, preferably between at least two calibration markers located on opposite sides of the track, wherein the virtual plane defines a timing line for the timing system; And saving the coordinates of the virtual plane in the memory of the timing system. 14. A computer-readable storage medium on which is stored at least one piece of software that, when executed or processed by a computer, is configured to perform the steps of the method according to any one of claims. 1-9.