WO2016007988A1

WO2016007988A1 - Systems for reviewing sporting activities and events

Info

Publication number: WO2016007988A1
Application number: PCT/AU2015/000407
Authority: WO
Inventors: Warren Brennan
Original assignee: Brennan Broadcast Group Pty Ltd
Priority date: 2014-07-16
Filing date: 2015-07-15
Publication date: 2016-01-21
Also published as: AU2015291766A1

Abstract

The system for reviewing incidents in sporting activities uses a sound detector (24) carried by a moving carrier, particularly an umpire (20, 21), to detect predetermined sounds produced by agents (ball 11, goal post 12, timer's siren) external to the moving carrier during the course of the sporting activity and to transmit the sound data signals for remote recording. A video image capture and recording system (35) captures and records sequences of images of the sporting activity for later retrieval and replay. A replay system (40) retrieves video image data and present by means of a display (46) a sequence of images around the time of an incident of interest, and also generates and manifests also by means of the display (46) sound information in synchronism with the sequence of images around the time of the incident of interest. Time shifter (50) corrects for different time of travel of the predetermined sound to the detector compared to instantaneous video image capture.

Description

SYSTEMS FOR REVIEWING SPORTING ACTIVITIES AND EVENTS

Field of the invention

This invention relates to systems for reviewing sporting activities and events using video image capture and display.

Background

There are numerous types of incidents in sporting activities or events where a particular sound can provide information to participants, officials, coaching or training personnel, commentators, and perhaps even medical personnel.

For example, in Australian Rules Football, if the ball, after being kicked by a player, is touched before the ball is then caught by another player, under the rules of the game, the play is to continue whereas if the ball had not been touched, the player who caught the ball would be awarded a "mark" and entitled to take possession of the ball and kick the ball without interference by an opponent. The momentary sound when the ball is touched is frequently detected by the officiating umpire who immediately calls "play on" to indicate to nearby players that any player catching the ball will not be awarded a mark.

Another example in Australian Rules Football of a sound being produced which can affect the way the rules of the game are applied occurs when the ball touches a goal post. Even if the ball travels in between the two upright goal posts, if it has touched one of the posts in doing so, a goal is not awarded under the rules of the game, but rather a "behind" is awarded. Sometimes the sound of the ball momentarily touching the post is detected by a goal umpire and/or by the principal central officiating umpire and the sound can confirm the impact even if there was some uncertainty from the visual monitoring of the ball movement during the course of the incident within the game.

A yet further example of sound being detected to confirm a relevant contact can arise when the ball having been kicked towards the goal is touched by a player before the ball passes in between the goal posts. If the ball has been touched, a "behind" is awarded, not a "goal". Again officials report using audible cues to help and sometimes even to determine whether the ball has been touched before passing between the goal posts.

In all of these kinds of incidents, television images captured during the play of the game are frequently used to attempt to determine if a relevant contact has occurred or not. Sometimes video images are being captured by high speed cameras to be recorded and selectively replayed in "slow motion replays". In particular, the captured images can be replayed at a slower frame rate than the capture rate so that the movement of the ball can be observed carefully to determine if a relevant momentary touch or contact occurred. Such reviews using slow motion replays have been most frequently used for commentary by the broadcaster for the information and entertainment of the viewers. Occasionally however, co-operative arrangements have been made between the sporting organisation conducting the sporting activity and the televising broadcaster whereby the slow motion replay can be presented to officials shortly after the incident of interest enabling a more reliable application of the rules of the game, e.g. if the ball was touched as evidenced by the replay but this was not detected by the officiating umpire during the actual play of the game. However such video replays, even when presented as slow motion replays, are frequently still ambiguous or uncertain whether an incident affecting application of the rules of the game has or has not occurred.

In cricket, a glancing contact of the ball with the bat or gloves of the batsman is frequently critically important because the batsman is dismissed if the ball is then caught by the wicket keeper or a fieldsman. This situation has been addressed by a system known as

"Snickometer"™. A fixed microphone in or adjacent the stumps detects nearby sounds such as the ball contacting the bat and by replaying these detected sounds, in synchronism with captured and recorded and then replayed video images, the occurrence of a sound indicative of the ball touching the bat or touching the gloves can be assessed. This Snickometer™ system has been described in patent specifications AU- 199952942 and GB-2457674 in the name of Plaskett. This system however may not be satisfactory for other sporting activities such as the circumstances in Australian Rules Football outlined above.

There are likewise incidents in other sporting codes where even slow motion replays of captured video images cannot determine whether a relevant contact has occurred. One example is in baseball where a glancing contact of the ball with the bat before the ball is caught by the catcher. This is a "foul tip" under the rules of baseball and audible cues can be used by the nearby referee to accurately identify such an occurrence.

A further significant incident or event in a sporting activity involving sound is the audible signal from a timer (including a mechanical or electrical timer and/or a person officiating as a time keeper or other official) outside the field of play initiating generation of a sound to signal termination of a period of play, or a suspension of play or "time out". Under the rules of some sports, such as Australian Rules Football, the central officiating umpire or referee upon hearing the siren in required to call and signal the end of play. Some events occurring before the moment the siren is started may be legitimately completed or concluded under the relevant rules, whereas events after the initiation and umpire's detection of the siren are deemed after the end of play. For example, the ball kicked before the siren is allowed to continue its flight or travel so that if it crosses the goal line during that travel, the relevant score is allowed, whereas a ball kicked even a fraction of a second after the start of the siren is not legitimately in play and cannot create a further score. Similarly, a football player who marked the ball before the commencement of the siren is permitted to exercise the right to utilise the kick of the ball, whereas a mark taken after commencement of the siren is deemed after the end of play. It can be difficult for the central officiating umpire to determine whether certain events occurred before or after the commencement of the siren particularly due to substantial ambient noise during the game and also the umpire's focus on the sporting activity occurring whereby the umpire may not immediately be conscious of the siren being sounded. Apart from television broadcasters and commentators replaying the captured video and audio signals and comparing the commencement of the siren or perhaps monitoring a superimposed time clock on the video images, there is no less subjective way of assessing whether events occurred before or after commencement of the siren.

Objects of the invention

It is an object of the present invention to provide a new or improved system (which includes apparatus and methods) for reviewing incidents in sporting activities utilising replay of captured video sequences.

It is a preferred and further object of the present invention to provide a new improved system for reviewing incidents in sporting activities and making a decision or making a ruling or providing a commentary concerning the incident of interest and application of the rules of the sporting activity relevant to that incident.

A further object of the invention according to the second aspect is to provide a system for reviewing incidents in sporting activities or events enabling a precise determination during a video replay of the sporting activity of the moment that the sporting activity terminates or is suspended.

Summary of the invention

According to a first aspect of the present invention, there is provided apparatus for reviewing incidents in sporting activities and events, the apparatus including:

a sound detector operative to detect predetermined sounds during the course of the sporting activity, and having a sound signal generator to generate sound data signals in response to detection of the sounds, and a transmitter operative to transmit the sound data signals for remote reception;

a sound signal receiver operative to receive the sound data signals from the transmitter; a sound data recorder operative to record sound data for the detected predetermined sounds in a period of time during which an incident of interest occurs; a video image capture and recording system operative to capture sequences of images of the sporting activity and to record the video image data for later retrieval and replay;

a replay system including a display, the replay system being operative to:

retrieve from the video recording system the recorded video image data and to present (by means of the display) a sequence of images around the time of an incident of interest, and

generate and manifest (by means of the display) sound information in synchronism with the sequence of images around the time of the incident of interest, the sound information being generated from the recorded sound data;

wherein the sound detector is adapted to be carried by a moving carrier generally following the sporting activities and wherein the predetermined sounds are those produced by agents external to the moving carrier.

By providing that the sound detector is carried by a moving carrier generally following the sporting activities, the sound detector can be on average nearer to incidents of interest occuning during the sporting activity than any fixed point in the vicinity of the location of the sporting event. For example, in one possible embodiment, the moving carrier is a motorised carrier to which the sound detector is mounted and which is movable so as to be closer to incidents occurring during the sporting activity than any fixed point in the vicinity of the location of the sporting event. The motorised carrier may be a carriage movable along a track provided inside the boundary fence of the sporting arena so as to be as close as possible to the events or actions occurring in the sporting activity.

However in a preferred embodiment, he moving earner may be an officiating person who is present during and who is following the sporting activities, such as a central or principal referee or umpire, a linesman or boundary line referee, goal umpire. The apparatus may include multiple sound detectors carried by multiple moving carriers, such as by multiple referees or umpires, wherein the sound data signals of the respective sound detectors can be automatically discriminated from each other, e.g. by having transmitters respectively operating at different transmission frequencies or by digitising the detected sounds and transmitting the digitized sound data signals with unique header information enabling differentiation, and the replay system is operative to enable replay of the video images with a selected one of the multiple sound data recordings.

In the sporting activities where the invention can be anticipated to be useful, the agents external to the moving carrier which produce the predetermined sounds are preferably selected from: contestants participating in the sporting activity, namely the players and their clothing or other items being worn,

devices being used by contestants participating in the sporting activity, such as a sports ball, bat, racquet, club, and

apparatus used in association with the sporting activity, e.g. goal or other marker posts, net, goal mouth top rail, boundary fence or wall or rail.

Preferably the apparatus, and particularly the method, for reviewing incidents in sporting activities and events, further includes performing an analysis of the output of the replay system, the analysis comprising comparing the synchronicity of displayed video images of a possible incident of interest which produces a characteristic sound which comprises the predetermined sound with the manifestation of sound information relating to sounds detected at the time of the possible incident of interest so as to make a decision or make a ruling or provide a commentary concerning the incident of interest and concerning the application of the rules of the sporting activity relevant to that incident.

The apparatus may further include a time shifter operative to relatively time shift the recorded video image data and the sound data so that when the sequences of images are presented by means of the display and the sound information is manifested by means of the display they are better synchronised than the actual video capture times and the sound detection times. Preferably the time shifter is operative to receive an input of separation data indicative of the spatial separation of the sound detector from the source of the predetermined sound from a possible incident of interest and, in response thereto, to calculate a time shift by which a manifestation of sound information is temporally advanced relative to the captured video frames around the time of the possible incident of interest.

In one embodiment, the input of separation data is a measure of the distance from the sound source to the sound detector, and the input is provided by an operator viewing the sporting activity, e.g. viewing a video replay around the time of the possible incident of interest, and inputting to the time shifter an estimated measure of the separation distance. In an alternative embodiment, the input of separation distance from the sound source to the sound detector is performed by an operator inputting, e.g. by operation of a pointer to indicate on a visual display of the sporting activity at the moment of the possible incident of interest, the location of the possible incident of interest and also the location of the sound detector and, from these two locations, computing a separation distance used to calculate the time shift. In one embodiment of the system, there is at least one tracker operative to provide location data indicative of at least one of:

• the location of the sound detector within the field of play of the sporting activity, and

• the location of a sports ball during the sporting activity within the field of play of the sporting activity;

the tracker being selected from:

• a GPS location signalling device, e.g. borne by the moving carrier of the sound detector, thereby providing location data for the sound detector,

• an object recognition system operative to identify and track movements within the field of sporting activity of the moving carrier with the sound detector and/or the sports ball being used in the sporting activity,

• an operator input system by which an operator who is monitoring a video replay of the spoiling activity inputs for successive frames of the recorded video images the instantaneous location within multiple frames of the sound detector and/or the sports ball being used in the sporting activity, and

and in this embodiment the separation distance from the sound source to the sound detector is automatically computed utilising the location data from the tracker.

According to a second possible use of apparatus of the present invention for reviewing incidents in sporting activities and events, the sound detector is adapted to be carried by a principal referee or umpire who has the responsibility to end a period of play or initiate a suspension of play in the sporting activity and the agent producing the predetermined sound external to the moving carrier comprises a timer operative to remotely generate an official audible signal to indicate the end of a period of play or to indicate suspension of play in the sporting activity.

In this second possible use, the manifested sound information may comprise a visible marker generated and presented by the display together with the video frame first captured at or after detection of the official audible signal. This enables in a replay clear visual indication of the instant when the officiating umpire is required by the rules of the sporting activity to call an end to play (regardless of whether the umpire actually heard the signal and reacted as required).

The invention in a second aspect also provides a method for reviewing incidents in sporting activities and events, the method including: providing a sound detector and detecting predetermined sounds during the course of the sporting activity, the sound detector having a sound signal generator generating sound data signals in response to detection of the sounds, and a transmitter transmitting the sound data signals for remote reception;

providing a sound signal receiver receiving the sound data signals from the transmitter; providing a sound data recorder recording sound data for the detected predetermined sounds in a period of time during which an incident of interest occurs;

providing a video image capture and recording system capturing sequences of images of the sporting activity and recording the video image data for later retrieval and replay;

providing a replay system including a display, the replay system being operated to:

retrieve from the video recording system the recorded video image data and present (by means of the display) a sequence of images around the time of an incident of interest, and

wherein the sound detector is carried by a moving earner generally following the sporting activities and wherein the predetermined sounds are those produced by agents external to the moving earner,

Preferred and optional features or steps of the method according to the second aspect of the invention will be understood from the preceding summary of features of the apparatus.

The invention in a third aspect also provides data files produced by the apparatus of the first aspect or by the method of the second aspect of the invention and capable of being transmitted, stored, and selectively replayed to reproduce the video sequences and associated sound manifestations.

According to a fourth aspect of the present invention, there is provided a system for reviewing incidents in sporting activities and events, the system including:

a sound detector carried by an officiating person who is present during and who is following the sporting activities, such as a central or principal referee or umpire, the sound detector being operative to detect sounds during the course of the sporting activity, and having a sound signal generator to generate sound data signals in response to detection of the sounds, and a transmitter operative to transmit the sound data signals for remote reception;

a sound signal receiver operative to receive the sound data signals from the transmitter; a sound data recorder operative to record sound data for the detected sounds in a period of time during which an incident of interest occurs;

video image capture and recording system operative to capture sequences of images of the sporting activity and to generate and record the video image data for later retrieval and replay; a replay system including a display, the replay system being operative to:

wherein the system includes a game timing input to the replay system providing a timing signal indicative of the moment that the game play is ended or suspended, e.g. a signal directly from the official timekeeping output or a signal from a sound detector in the vicinity of the official time sounding siren or the like, the replay system being operative in response to the timing signal to generate and present by the display a visible time marker simultaneously with the presentation (by means of the display) of the sequence of images around the time of an incident of interest and simultaneously with the manifestation (by means of the display) of the sound information in synchronism with the sequence of images around the time of the incident of interest, and wherein the actual moment of presentation of the displayed visible time marker is in exact synchronism with the video images captured independent of the sound data signals (which may be relatively time shifted before presentation of the displayed sound information generated therefrom).

According to a fifth aspect of the present invention, there is provided a system for reviewing incidents in sporting activities and events, the system including:

a sound signal receiver operative to receive the sound data signals from the transmitter;

a sound data recorder operative to record sound data for the detected sounds in a period of time during which an incident of interest occurs; video image capture and recording system operative to capture sequences of images of the sporting activity and to generate and record the video image data for later retrieval and replay; a replay system including a display, the replay system being operative to:

wherein the system includes a game timing input of the replay system providing a timing signal indicative of the moment that the game play is ended or suspended, e.g. a signal directly from an official timekeeping output or a signal from a sound detector in the vicinity of the official time sounding siren or the like, the replay system being operative in response to the timing signal to generate and present by the display a visible time marker simultaneously with the presentation (by means of the display) of the sequence of images around the time of an incident of interest and simultaneously with the manifestation (by means of the display) of the sound information in synchronism with the sequence of images around the time of the incident of interest, and wherein the system further includes a time shifter operative to relatively time shift the display of the visible time marker so that it is delayed relative to the video images captured at the moment of provision of the timing signal indicative of the moment that the game play is ended or suspended by an interval determined or calculated to correspond to the time of travel of the sound from the official time sounding siren or the like to the officiating person carrying the sound detector (whereby the time marked is displayed in synchronism with the image at the time the siren sound would have reached to the officiating person). This embodiment is useable in sporting activities having rules which require a main or central official to receive externally generated the sound and immediately terminate the play.

Throughout this specification where there are references to the invention using the term "system", it is intended and to be understood that that term encompasses both methods and apparatus.

Summary of the drawings

Possible and preferred features of the present invention will now be described with particular reference to the accompanying drawings. However it is to be understood that the features illustrated in and described with reference to the drawings are not to be construed as limiting on the scope of the invention. In the drawings: Fig. 1 is a schematic illustration of an apparatus according to the invention in use during an Australian Rules Football game,

Fig. 2 is a schematic illustration of a possible display being presented by a replay system enabling analysis of an incident of interest.

Description of preferred embodiments

The drawings schematically illustrate a system in use for reviewing incidents occurring in an Australian Rules Football game being played on the playing field 10. A single incident in the game is depicted involving a player 15 possibly marking or touching the football 1 1 in close proximity to a goal post 12 and close to the boundary or goal line 13. An umpire 20, who may be a central officiating umpire, or may be a boundary umpire, is watching the incident from a position near the behind post 14 which is spaced 6.4 metres from the goal post 12. A goal umpire 21 is also observing the incident, but his view of the ball 1 1 at that instant may be obscured by the goal post 12 and hence the goal umpire 21 may be unable to observe whether the ball 1 1 touches the goal post 12 or whether the player 15 has touched the ball 1 1.

The system of the invention includes a sound detector 24 carried by the umpire 20 who, during the game, follows the activities so that the sound detector 24 will be on average nearer to incidents of interest (i.e. the ball 11 contacting the goal post 12 and/or contacting the player 15 in the incident depicted) than any fixed point in the playing field 10 or nearby. Although the sound detector is depicted being carried by umpire 20, the system could also comprise a moving carrier generally following the sporting activities in other ways, such as a sound detector mounted by a carriage moving in a track around the boundary fence or the boundary line of the playing field 10, or perhaps a sound detector carried by an overhead carrier being driven by means of an overhead suspension system and following the sporting activity, e.g. of the kind used for overhead video cameras used at some sporting stadiums and which can be remotely controlled to travel over or hover over the field of play.

The goal umpire 21 can also carry a similar sound detector 25. Thus there can be multiple sound detectors carried by multiple moving or moveable carriers. It may also be possible to provide game players with respective sound detectors. The or each sound detector 24, 25 may be of generally conventional construction and operation, having a sound sensitive member such as a microphone or transducer to detect sounds and a sound signal generator to generate sound data signals, and having a transmitter operative to transmit the sound data signals for remote reception, e.g. by RF transmissions. Microphones carried by umpires at sporting events are known so that their speech and the speech of participating players can be detected and transmitted to a remote reception, e.g. for entertainment or information purposes. Where multiple sound detectors 24, 25 are used, discrimination between the respective sound detectors may be enabled by having the sound detectors transmit at different frequencies or by digitising the sound data and providing unique metadata or header information enabling differentiation of digitised sound data signals transmitted by the different sound detectors.

The system includes a sound signal receiver 30 to receive the sound data signals from the transmitter of the sound detector 24, 25. This can be a conventional RF antenna and associated receiver circuitry such as filters, tuners, amplifiers, etc. Associated with the receiver 30 is a sound data recorder 34 which records sound data for the detected sounds, particularly during a period of time during which an incident of interest occurs. The sound data recorder 34 may be generally conventional and may comprise for example digital processing means which controls storage or recording of sound data with associated information concerning the particular sound detector 24, 25 from which the data was received, timing information, and other data enabling later selective retrieval.

The system also includes a video capture and recording system 35 operative to capture sequences of images of the sporting activity and to generate and record the video image data for later retrieval and replay. This system 35 may be generally conventional in construction and operation. For example, television cameras 36, 37 may be operated to view the playing field 10 from various directions. These cameras 36, 37 can be conventional television cameras operating at standard frame capture rates (25 frames per second). Preferably some or all of the cameras are high speed cameras enabling slow motion replays, conventionally capturing frames at 75 per second. Very high speed television cameras are now available enabling frame capture rates of 150 frames per second and video sequences captured and stored from such cameras can be used in "super slow motion" replays. The captured sequences of images from the cameras 36, 37 can be processed and stored conventionally by video processor and store 38 enabling later selective retrieval and replay.

The replay system 40 includes a display 45. The replay system 40 is operative to retrieve from the video recording system 35, the recorded video image data and to present by means of the display 45 a sequence of video images 46 around the time of the incidents of interest. As mentioned above, these can be slow motion or super slow motion replays. The replay system 40 also operates to generate and manifest by means of the display 45, sound information 47 in synchronism with the sequence of images 46 around the time of the incident of interest, the sound information 47 being generated from recorded sound data from the recorder 34. In the case where multiple sound detectors 24, 25 are in use and respective sound data from the multiple detectors has been recorded, the sound information 47 manifested by the display 45 may be derived from either of the detectors under selective operator control so that the respective detector's sound data yielding a clearer manifestation 47 at the time of the incident of interest can be used for display, analysis, commentary, ruling, etc. The sound information 47 which is schematically illustrated, comprises a trace of the amplitude of the sound detected leading up to, during, and shortly after the incident of interest. The trace may comprise a line graph with sound amplitude on the vertical axis and time across a horizontal axis. The sound trace may be processed by the replay system to enhance its clarity, e.g. by filtering, amplification and/or clipping, active background noise cancellation, colour manipulation to enhance parts of the sound trace identified to be substantially at the moment of the incident of interest, etc. During the replay of the video images 46 and the presentation or manifestation of the sound data 47, the point of time along the horizontal time axis of the sound trace corresponding to the displayed video frame can be highlighted by a moving light point along the trace, or by a coloured vertical bar moving along the horizontal axis, or other means. Another possible way of manifesting sound information is to provide a frequency analyser display (similar to those used in music replay systems where bar graphs operating for different frequency bands rise and fall to represent the sound output within the respective frequency bands). Such a frequency spectrum can produce on the display a time varying manifestation before, during, and after the corresponding replay of the video images.

The process of the invention may include the further step of performing an analysis of the output by the display 45 of the replay system, the analysis comprising comparing the features (including the synchronicity) of displayed video images 46 of a possible incident of interest which produces a predetermined or characteristic sound, and the manifestation 47 of sound information relating to sounds detected at the time of the possible incident of interest so as to make a decision or make a ruling or provide a commentary concerning the incident of interest and application of the rules of the sporting activity relevant to that incident . The analysis may be performed by an operator, a viewer, an umpire or referee, a commentator, etc. The features compared can include whether a sound has been detected, what was the nature or were the properties of that sound, when the sound was produced (compared to the video images of the incident), and determination or inference of the cause of the sound. The applicability of certain rules of the sport can then be decided.

The system further includes a time shifter 50 operative to relatively time shift the recorded video image data and the sound data so that when the sequences of images are presented by means of the display 45 and the sound information is manifested at 47 by means of the display 45 they are better synchronised than the actual video capture times and the sound detection times. Since sound travels at a relatively slow speed (compared to light being captured by the cameras), the sound manifestation may be advanced and displayed earlier than the image captured at the same instant, or vice versa by delay of the image displayed.

To explain this problem of synchronising the recorded sound with the captured video images, the speed of sound in air and its influence on the data capture needs to be considered. The speed of sound in air is approximately 343m per second. The standard television frame capture rate for most purposes has been 25 frames per second ("fps"). For slow motion replays of events or activities in which there is considerable movement, the frame capture rate is 75 fps. Thus when a sequence of video frames captured at 75fps is replayed at the standard screen refresh rate of 25fps, the motion is "slowed" to one third of the actual speed. Advances in camera technology have now enabled a "super slow motion" video camera to be available. This "super slomo" camera captures frames at 1 SOfps. Using this frame capture speed of 150fps, between successive frames (ie. in 0.0067 seconds), sound travels about 2.3m. This means that if a sound from an incident of interest occurs 2.3m from the sound detector canied by an umpire 20 or 21 , the sound will be detected one complete video frame interval later than a frame captured at the instant of the sound generation. In the drawings, the umpire 20 is about 8 metres from the ball 11 so sound from a contact of the ball 1 1 with the post 12 or with a hand of the player 15 will reach the detector 24 after three captured video frames have been recorded and stored from a "super slomo" camera, or after one video frame has been recorded and stored from a camera capturing images at 75fps. Thus without the time shifter 50m operating there may be a misleading disjunction between sound from an incident of interest in the sporting activity and its slow motion video replay, the degree of disjunction being dependent on the (variable) separation of the sound detector and the source of the sound of interest.

In the preferred illustrated system, the time shifter 50 is operative to respond to an input of separation data indicative of the spatial separation of the sound detector 24, 25 from the source of sound from a possible incident of interest and, in response, thereto to calculate a time shift by which the manifestation 47 of sound information is temporally advanced relative to the captured video frames around the time of the possible incident of interest. The input of separation data is a measure of the distance from the sound source (ball 1 1) to the particular sound detector 24, 25 being used for the audio input. The input is provided by an operator viewing the sporting activity, e.g. viewing the video replay images 46 around the time of the possible incident of interest, and inputting an estimated measure of the separation distance.

In one possible embodiment, the input of separation distance from the sound source (ball 1 1) to the selected sound detector 24 or 25 may be performed by an operator inputting, e.g. by operation of a pointer such as a cursor on the display 45 moved by operation of mouse 52, or by indicating on a touch screen of the display 45, to indicate on the visual display 46 of the sporting activity at the moment of the possible incident of interest the location of the possible incident of interest (e.g. ball 1 1 touching the post 12 or the player's hand) and also the location of the sound detector 24 and, from these two locations, computing a separation distance used to calculate the time shift.

In an alternative system, at least one tracker can be operative to provide location data indicative of at least one of: (a) the location within the of field of play 10 of the sporting activity of the sound detector 24, and (b) the location within the field 10 of the sporting activity of the sports ball 11 during the sporting activity. The tracker can be selected from:

(a) a GPS location signalling device, e.g. borne by the moving carrier (umpire 20) of the sound detector 24, thereby automatically continuously providing location data for the sound detector,

(b) an object recognition system operative to identify and track movements within the field of sporting activity of the moving carrier (umpire 20) with the sound detector 24 and/or the sports ball 1 1 being used in the sporting activity,

(c) an operator input system 53 by which an operator who is monitoring the video replay 46 of the sporting activity inputs for successive frames of the recorded video images the instantaneous location within multiple frames of the sound detector 24 and/or the sports ball 1 1 being used in the sporting activity.

Using such a tracker the separation distance from the sound source to the sound detector is automatically computed by appropriate processor 54 utilising the location data from the tracker.

In another aspect of the invention the system for reviewing incidents in sporting activities and events includes the sound detector 24 carried by the officiating person (particularly the central or principal referee or umpire) and operative to detect sound of an official audible signal, such as a time keeper's signal or siren indicating the end of a period of play, or suspension of play. The sound detector 24 may be constructed, or the sound data therefrom may be processed e.g. by filtering, to be particularly sensitive to the frequency of the official audible signal. If desired the official audible signal may include a sound component which has a frequency beyond human hearing range, e.g. ultrasonic, and the sound detection may have a tuned response to that frequency to reliably identify the reception at the detector 24 of the official audible signal. As a result of this processing, that when the replay system 40 manifests the detected and recorded sound information, the moment of detection by the sound detector 24 carried by the officiating person 20 of the official audible signal can be clearly visible e.g. by being specially marked in the displayed manifestation 47. For example, the manifested sound information may comprise a visible marker generated and presented by the display 45 together with the video frame first captured at or after detection of the official audible signal. Thus the exact time in the replayed sequence of video images of the sports play can be seen when the officiating person 20 received the time keeper's signal or siren indicating the end of a period of play, or suspension of play. The replay can therefore resolve exactly when in the sporting activity the rules of the sport concerning valid plays became operative.

In general it will be seen that the embodiments of the invention enable the sound detector adapted to be carried by a moving carrier generally following the sporting activities to detect several types of predetermined sounds which are produced by agents external to the moving carrier. Such agents can include contestants participating in the sporting activity (e.g. player 15 and boots, clothing worn thereby), devices being used by contestants participating in the sporting activity (e.g. ball 1 1), and apparatus used in association with the sporting activity (e.g. goal post 12, behind post 14). The other possible external agent described above comprises the timer producing an official audible signal to indicate end of a period of play. However other external agents in other sports can be involved in the production of predetermined sounds which, upon detection, can require the application of particular rules governing the sporting activity.

When used in this specification and claims, the terms "comprises" and "comprising" and variations thereof mean that the specified features, steps or integers are included. The terms are not to be interpreted to exclude the presence of other features, steps or components.

It is to be understood that various alterations, modifications and/or additions may be made to the features of the possible and preferred embodiment(s) of the invention as herein described without departing from the scope of the invention as defined in the appended claims.

Claims

1. Apparatus for reviewing incidents in sporting activities and events, the apparatus including:

a sound signal receiver operative to receive the sound data signals from the transmitter; a sound data recorder operative to record sound data for the detected predetermined sounds in a period of time during which an incident of interest occurs;

a video image capture and recording system operative to capture sequences of images of the sporting activity and to record the video image data for later retrieval and replay;

a replay system including a display, the replay system being operative to:

2. Apparatus as claimed in claim 1 wherein the moving carrier is an officiating person who is present during and who is following the sporting activities such as a central or principal referee or umpire, a linesman or boundary line referee, a goal umpire.

3. Apparatus as claimed in claim 1 wherein the moving carrier is a motorised carrier to which the sound detector is mounted and which is movable so as to be closer to incidents occurring during the sporting activity than any fixed point in the vicinity of the location of the sporting event.

4. Apparatus as claimed in claim 2 wherein the apparatus includes multiple sound detectors carried by multiple moving carriers, such as by multiple referees or umpires, wherein the sound data signals of the respective sound detectors can be automatically discriminated from each other, e.g. by having transmitters respectively operating at different transmission frequencies or by digitising the detected sounds and transmitting the digitized sound data signals with unique metadata or header information enabling differentiation, and wherein the replay system is operative to enable replay of the video images with a selected one of the multiple sound data recordings.

5. Apparatus as claimed in any one of the preceding claims wherein the agents external to the moving carrier which produce the predetermined sounds are selected from:

contestants participating in the sporting activity,

devices being used by contestants participating in the sporting activity, and

apparatus used in association with the sporting activity.

6. Apparatus as claimed in claim 5 and further including performing an analysis of the output of the replay system, the analysis comprising comparing the synchronicity of displayed video images of a possible incident of interest which produces a characteristic sound which comprises the predetermined sound with the manifestation of sound information relating to sounds detected at the time of the possible incident of interest so as to make a decision or make a ruling or provide a commentary concerning the incident of interest and concerning the application of the rules of the sporting activity relevant to that incident.

7. Apparatus as claimed in any one of the preceding claims and further including a time shifter operative to relatively time shift the recorded video image data and the sound data so that when the sequences of images are presented by means of the display and the sound information is manifested by means of the display they are better synchronised than the actual video capture times and the sound detection times.

8. Apparatus as claimed in claim 7 wherein the time shifter is operative to receive an input of separation data indicative of the spatial separation of the sound detector from the source of the predetermined sound from a possible incident of interest and, in response thereto, to calculate a time shift by which a manifestation of sound information is temporally advanced relative to the captured video frames around the time of the possible incident of interest.

9. Apparatus as claimed in claim 8 wherein the input of separation data is a measure of the distance from the sound source to the sound detector, and wherein the input is provided by an operator viewing the sporting activity, e.g. viewing a video replay around the time of the possible incident of interest, and inputting to the time shifter an estimated measure of the separation distance.

10. Apparatus as claimed in claim 8 wherein the input of separation distance from the sound source to the sound detector is performed by an operator inputting, e.g. by operation of a pointer to indicate on a visual display of the sporting activity at the moment of the possible incident of interest, the location of the possible incident of interest and also the location of the sound detector and, from these two locations, computing a separation distance used to calculate the time shift.

1 1. Apparatus as claimed in claim 8 wherein the system includes at least one tracker operative to provide location data indicative of at least one of:

· the location of the sound detector within the field of play of the sporting activity, and

the tracker being selected from:

• a GPS location signalling device, e.g. borne by the moving carrier of the sound

detector, thereby providing location data for the sound detector,

• an operator input system by which an operator who is monitoring a video replay of the sporting activity inputs for successive frames of the recorded video images tine instantaneous location within multiple frames of the sound detector and/or the sports ball being used in the sporting activity, and

wherein the separation distance from the sound source to the sound detector is automatically computed utilising the location data from the tracker.

12. Apparatus as claimed in claim 2 and wherein the sound detector is adapted to be carried by a principal referee or umpire who has the responsibility to end a period of play or initiate a suspension of play in the sporting activity and wherein the agent producing the predetermined sound external to the moving canier comprises a timer operative to remotely generate an official audible signal to indicate the end of a period of play or to indicate suspension of play in the sporting activity.

13. Apparatus as claimed in claim 12 wherein the manifested sound information comprises a visible marker generated and presented by the display together with the video frame first captured at or after detection of the official audible signal.

14. A method for reviewing incidents in sporting activities and events, the method including: providing a sound detector and detecting predetermined sounds during the course of the sporting activity, the sound detector having a sound signal generator generating sound data signals in response to detection of the sounds, and a transmitter transmitting the sound data signals for remote reception;

wherein the sound detector is carried by a moving carrier generally following the sporting activities and wherein the predetermined sounds are those produced by agents external to the moving carrier.

15. A method as claimed in claim 14 wherein the moving carrier is an officiating person who is present during and who is following the sporting activities such as a central or principal referee or umpire, a linesman or boundary line referee, a goal umpire.

16. A method as claimed in claim 15 wherein multiple sound detectors are carried by multiple moving carriers, such as by multiple referees or umpires, wherein the sound data signals of the respective sound detectors are automatically discriminated from each other, e.g. by having transmitters respectively operating at different transmission frequencies or by digitising the detected sounds and transmitting the digitized sound data signals with unique metadata or header information enabling differentiation, and wherein the replay system is operated to replay the video images with a selected one of the multiple sound data recordings.

17. A method as claimed in any one of claims 14 to 16 wherein the agents external to the moving carrier which produce the predetermined sounds are selected from:

contestants participating in the sporting activity,

devices being used by contestants participating in the sporting activity, and

apparatus used in association with the sporting activity.

18. A method as claimed in claim 17 and further including performing an analysis of the output of the replay system, the analysis comprising comparing the synchronicity of displayed video images of a possible incident of interest which produces a characteristic sound which comprises the predetermined sound with the manifestation of sound information relating to sounds detected at the time of the possible incident of interest so as to make a decision or make a ruling or provide a commentary concerning the incident of interest and concerning the application of the rules of the sporting activity relevant to that incident.

19. A method as claimed in any one of claims 14 to 18 and further including relatively time shifting the recorded video image data and the sound data so that when the sequences of images are presented by means of the display and the sound information is manifested by means of the display they are better synchronised than the actual video capture times and the sound detection times.

20. A method as claimed in claim 19 wherein the step of time shifting includes receiving an input of separation data indicative of the spatial separation of the sound detector from the source of the predetermined sound from a possible incident of interest and, in response thereto, calculating a time shift by which a manifestation of sound information is temporally advanced relative to the captured video frames around the time of the possible incident of interest.

21. A method as claimed in claim 20 wherein the input of separation data is a measure of the distance from the sound source to the sound detector, and wherein the input is provided by an operator viewing the sporting activity, e.g. viewing a video replay around the time of the possible incident of interest, and inputting for enabling the step of time shifting an estimated measure of the separation distance.

22. A method as claimed in claim 20 wherein the input of separation distance from the sound source to the sound detector is performed by an operator inputting, e.g. by operation of a pointer to indicate on a visual display of the sporting activity at the moment of the possible incident of interest, the location of the possible incident of interest and also the location of the sound detector and, from these two locations, computing a separation distance used to calculate the time shift.

23. A method as claimed in claim 20 wherein there is provided at least one tracker operating to provide location data indicative of at least one of:

• the location of a sports ball during the sporting activity within the field of play of the sporting activity; the tracker being selected from:

detector, thereby providing location data for the sound detector,

• an object recognition system operating to identify and track movements within the field of sporting activity of the moving carrier with the sound detector and/or the sports ball being used in the sporting activity,

• an operator input system by which an operator who is monitoring a video replay of the sporting activity inputs for successive frames of the recorded video images the instantaneous location within multiple frames of the sound detector and/or the sports ball being used in the sporting activity, and

24. A method as claimed in claim 14 and wherein the sound detector is being carried by a principal referee or umpire who has the responsibility to end a period of play or initiate a suspension of play in the sporting activity and wherein the agent producing the predetermined sound external to the moving carrier comprises a timer operative to remotely generate an official audible signal to indicate the end of a period of play or to indicate suspension of play in the sporting activity.

25. A method as claimed in claim 24 wherein the manifested sound information comprises a visible marker generated and presented by the display together with the video frame first captured at or after detection of the official audible signal.

26. Data files produced by the apparatus as claimed in any one of claims 1 to 13 or by the method as claimed in any one of claims 14 to 25 and capable of being transmitted, stored, and selectively replayed to reproduce the video sequences and associated sound manifestations.

27. A system for reviewing incidents in sporting activities and events, the system including: a sound detector carried by an officiating person who is present during and who is following the sporting activities, such as a central or principal referee or umpire, the sound detector being operative to detect sounds during the course of the sporting activity, and having a sound signal generator to generate sound data signals in response to detection of the sounds, and a transmitter operative to transmit the sound data signals for remote reception;

28. A system for reviewing incidents in sporting activities and events, the system including: a sound detector carried by an officiating person who is present during and who is following the sporting activities, such as a central or principal referee or umpire, the sound detector being operative to detect sounds during the course of the sporting activity, and having a sound signal generator to generate sound data signals in response to detection of the sounds, and a transmitter operative to transmit the sound data signals for remote reception;

a sound data recorder operative to record sound data for the detected sounds in a period of time during which an incident of interest occurs;

generate and manifest (by means of the display) sound information in synchionism with the sequence of images around the time of the incident of interest, the sound information being generated from the recorded sound data;

wherein the system includes a game timing input of the replay system providing a liming signal indicative of the moment that the game play is ended or suspended, e.g. a signal directly from an official timekeeping output or a signal from a sound detector in the vicinity of the official time sounding siren or the like, the replay system being operative in response to the timing signal to generate and present by the display a visible time marker simultaneously with the presentation (by means of the display) of the sequence of images around the time of an incident of interest and simultaneously with the manifestation (by means of the display) of the sound information in synchronism with the sequence of images around the time of the incident of interest, and wherein the system further includes a time shifter operative to relatively time shift the display of the visible time marker so that it is delayed relative to the video images captured at the moment of provision of the timing signal indicative of the moment that the game play is ended or suspended by an interval determined or calculated to correspond to the time of travel of the sound from the official time sounding siren or the like to the officiating person carrying the sound detector (whereby the time marked is displayed in synchionism with the image at the time the siren sound would have reached to the officiating person). This embodiment is useable in sporting activities having rules which require a main or central official to receive externally generated the sound and immediately terminate the play.