AU753622B2

AU753622B2 - Method of, and system for, analysing events

Info

Publication number: AU753622B2
Application number: AU52942/99A
Authority: AU
Inventors: Allan Plaskett
Original assignee: Individual
Current assignee: Individual
Priority date: 1998-08-12
Filing date: 1999-08-05
Publication date: 2002-10-24
Anticipated expiration: 2019-08-05
Also published as: WO2000010333A1; GB0102998D0; GB2358755A8; GB2358755A; GB2358755B; AU5294299A

Description

WO 00/10333 PCT/GB99/02573 1 METHOD OF, AND SYSTEM FOR, ANALYSING EVENTS The present invention relates to a method of, and a system for, analysing events. In particular, but not exclusively, the invention relates to a method of, and a system for, analysing the sequence of events as a batsman makes a stroke at the ball during a game of cricket.

The game of cricket is, of course, played by both men and women. For simplicity, in the following description the male gender is used throughout and is intended to include the female gender.

Slow motion television replays have for many years been used by commentators to analyse the decisions of cricket umpires, who have to make their decisions based on what they see and hear while standing at the wicket. This has led to many disputes as to whether the umpires' decisions have been correct. However, the ability of the television companies to provide these action replays is of considerable commercial importance to them, as it greatly enhances the viewer's entertainment and interest in the cricket match.

Recently, television replays have also been used for making the actual decision on appeals for run outs, the decision being made by a "third umpire" offthe pitch, who views video recordings of either the broadcast television picture or the pictures from dedicated television cameras.

The ability of the umpires to make correct decisions is of considerable commercial as well as sporting importance, given the very large sums of income from sponsorship and prize money that may depend on the outcome of important first class games, and the investment made by television companies for the right to broadcast important matches.

Two areas that can present difficulty for umpires are appeals for catches and 1.b.w. (leg before wicket), particularly when the ball passes the batsman and the umpire is unsure whether the ball has hit the bat. For a batsman to be given out caught, the ball must strike the bat (or the batsman's protective gloves), no matter how finely, and then be caught before hitting the ground. Although the ball may appear to be deflected as it passes the batsman, and although the umpire may hear a noise similar to that of the ball hitting the bat, he may be unsure whether the ball was deflected by the bat or by, for example, the batsman's pads or some other part of his body, and whether any sound he heard was the sound of the ball hitting the bat or the pads or, perhaps, the bat hitting the ground or the pads. Commentators frequently question the 2 umpires' decisions in such cases but, even with slow motion action replays, it may be difficult for them to judge whether the correct decision was reached.

As regards appeals for a batsman cannot be given out I.b.w. if the ball strikes his bat, again no matter how finely, before striking his pads. The umpire may be unsure whether the ball hit the bat or, if it hit both the bat and the pads, which it hit first. Again, even with slow motion action replays, it may be difficult to decide what happened.

It is an advantage of at least one embodiment of the present invention to provide method of, and a device for, analysing events that mitigates at least some of the afore-mentioned problems.

According to the present invention there is provided a method of analysing events, in which a television signal of an event to be analysed is recorded, the video part of the recorded signal is stored as a set of data files representing in a graphical format selected frames of the video signal, the audio part of the recorded signal is converted into a graphical format representing the waveform of the audio signal compensation is provided for any real time delay between the audio part of the recorded signal and the video part "of the recorded signal, and the graphical representation of the audio signal is 0presented with the time compensated frames of the video signal in a combined graphical format display, allowing the simultaneity of the audio and video events recorded in the television signal to be assessed.

The system allows events to be analysed by matching a sequence of television pictures frame-by-frame with a graphical representation of the sounds caused by those events. This makes it possible to ascertain exactly when an audible event took place, allowing it to be matched with the events that can be seen in the television pictures to have taken place at that moment.

Thus, in the example of a cricket match, it may be possible to judge whether an •audible "snick" was caused by the ball hitting the edge of the bat or something else, such as the bat or the ball hitting the batsman's pads.

:30 Advantageously, the video part of the recorded signal is stored as an AVI file. Advantageously, the frames of the video signal are stored as a set of DIB files.

Advantageously, the audio part of the recorded signal is stored as digital data in a text file.

Advantageously, digital data representing the audio part of the recorded signal is converted into a graphical format by a data analysis program.

Alternatively, the audio part of the recorded signal may be converted into a graphical format by a virtual oscilloscope device.

Advantageously, the graphical representation of the audio signal and the selected frames of the video signal are presented in the combined graphical format display within a graphical user environment.

Advantageously, the audio part of the recorded signal is output as an analogue audio output.

Advantageously, compensation is provided for any time delay between the audio part of the recorded signal and the video part of the recorded signal.

Advantageously, the audio part of the recorded signal is filtered.

Advantageously, the audio part of the recorded signal is analysed to determine the frequencies present in the audio part.

Advantageously, the frequency analysis is obtained by means of a fast Fourier transform.

Advantageously, the frequency analysis is presented graphically in the combined graphical format display.

Advantageously, the combined graphical format display is converted into :an AN output signal.

According to a further aspect of the invention there is provided a system for analysing events, said system including a television camera, a microphone, a recording device for recording a television signal of an event captured by the television camera and the microphone, and a data processing device connected to the recording device for analysing the recorded television signal, the data processing device being arranged to store the video part of the recorded signal as a set of data files representing in a graphical format selected frames of the video signal, to convert the audio part of the recorded signal into o a graphical format representing the waveform of the audio signal to compensate for any real time delay between the audio part of the recorded signal and the video part of the recorded signal, and to present the graphical 30 representation of the waveform of the audio signal with the time compensated :frames of the video signal in a combined graphical format display, allowing the simultaneity of the audio and video events recorded in the television signal to be assessed.

Advantageously, the video part of the recorded signal is stored as an AVI file.

WO 00/10333 PCT/GB99/02573 4 Advantageously, the frames of the video signal are stored as a set of DIB files.

Advantageously, the audio part of the recorded signal is stored as digital data text file.

Advantageously, the system includes a data analysis program for converting the digital data representing the audio part of the recorded signal into a graphical format.

Advantageously, the system includes a virtual oscilloscope device for converting the audio part of the recorded signal is converted into a graphical format.

Advantageously, the system includes means for compensating for any time delay between the audio part of the recorded signal and the video part of the recorded signal.

Advantageously, the system includes means for filtering the audio part of the recorded signal.

Advantageously, the system includes means for analysing the audio part of the recorded signal to determine the frequencies present in the audio part.

Advantageously, the frequency analysis means employs a fast Fourier transform process.

Advantageously, the output of the frequency analysis means is presented graphically in the combined graphical format display.

Advantageously, the system includes means for converting the combined graphical format display into an A/V output signal.

Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which: Fig. 1 is a block diagram of computer-based system for analysing events from a television signal; Fig. 2 is a block diagram of a computer that forms part of the system shown in Fig. 1; WO 00/10333 PCT/GB99/02573 Fig. 3 is a flow diagram illustrating the steps ofa process for analysing events using the system; Figs. 4a, 4b and 4c are successive selected video frames from a television broadcast, showing a sequence of events at a cricket match; Fig. 5 is a graphical representation of the waveform of the audio signal accompanying the selected video frames shown in Figs. 4a, 4b and 4c; Fig. 6 is a graphical representation of the frequency analysis of a first event depicted in the selected video frames; Fig. 7 is a graphical representation of the frequency analysis of a second event depicted in the selected video frames, and Fig. 8 is a block diagram of an alternative computer-based system for analysing events.

Fig. 1 is a block diagram of computer-based system for analysing events at, for example, a cricket match. The system includes at least one television camera 2 and at least one microphone 4, although more cameras and microphones may also be provided. The television camera and the microphone may form part of the television system used by a broadcaster for broadcasting coverage of the cricket match, or they may be dedicated devices that are used only within the analysis system. The microphone 2 is preferably positioned so that it will be close to the events that are to be analysed. For example, at a cricket match, the microphone is preferably mounted inside one of the stumps, or alternatively just behind the wicket.

The television camera 2 has an output 6 through which the video signal passes and the microphone 4 has an output 8 for the audio signal. These outputs are connected to a video recorder 10, which is preferably capable of super slow motion recording. The signals are combined by the video recorder to generate a synchronised composite audiovisual signal, which leaves the video recorder through an A/V output terminal 12.

The A/V output signal may be passed directly to the broadcaster for live transmission, or it may be recorded by the video recorder 10 for re-transmission later, for example in an action replay.

The A/V output terminal 12 of the video recorder is connected to a data processing device 14, for example a personal computer (PC) having a sound card and a video card. The video card allows audio-visual signals to be captured and stored as digital data in an audio-video WO 00/10333 PCT/GB99/02573 6 interleaved (AVI) file. Numerous suitable video cards are commercially available, one such being the card sold under the brand ALL-IN-WONDER PRO by ATI. This card is capable of capturing a television video signal at a rate of up to 25 frames per second and an audio signal at a sampling frequency of 11.025 kHz, 22.050 kHz or 44.100 kHz. The computer also includes a monitor 16 and an control device 18, for example a keyboard and/or a mouse, as well as the normal components ofa personal computer, such as a central processor unit (CPU), random access memory (RAM), read only memory (ROM) and so on (not shown).

The computer 14 has an video output terminal 20, for providing a video output signal illustrating the result of the analysis performed by the computer. This output video signal may be passed to the broadcaster, for transmission to the public, or it may be transmitted for viewing by the "third umpire", to enable him to make a decision with regard to a contested appeal. The computer 14 also has an audio output terminal 22, for providing an analogue audio output signal of the sounds associated with the visual events analysed by the computer.

Fig. 2 is a block diagram of computer forming part of the system shown in Fig. 1. The computer 14 has a video capture card 24 that receives the composite audio/video signal through an input terminal 26. The video card 24 captures the television signal and stores it as digital data in an audio-video interleaved (AVI) file. This captured AVI file leaves the video card through a first output terminal 28. The video card also has a second output terminal for direct output of video and audio signals.

The computer includes a software AVI analysis program 32, which receives the captured AVI file and splits the file to extract the separate video frames making up the video clip as a sequence of device independent bitmap (DIB) files, DIB(1) to DIB(n). The AVI analysis program 32 may for example be written as a Visual C++TM routine. Such a routine is available from the Internet as freeware from Phade Software and is called AVI RIP. The audio data is also extracted and stored as a set of 16-bit data points in a text (TXT) file.

The computer 14 also includes a software data analysis program 34 such as the spreadsheet program sold under the trade mark MICROSOFT EXCEL. Bespoke software 35 (including macros and other routines) is provided to control the data analysis program. This program receives the DIB files and displays the frames represented by those files, under control of an operator, in a window of a graphical user interface display (GUI) 36. The concatenated digital WO 00/10333 PCT/GB99/02573 7 audio signal file is also converted by the program into a graphical form representing the amplitude of the audio signal against time, which is displayed in a separate window in the GUI.

The GUI display 36 including the video and audio windows is converted by the video card 24 into a video output signal, which exits through the video output terminal 20. An analogue audio output signal is provided through the audio output terminal 22.

The process for analysing events using the system will now be described with reference to Fig.

3. The process has three main steps, these being: 1. Capture of the video signal and storing of the digital data representing that signal as an audio-video interleaved (AVI) file; 2. Analysis of the AVI file and separation of the audio and video data into separate files, and 3. Graphical presentation of video signal as a sequence of selected frames, and the audio signal as a waveform, illustrating the relationship between the recorded sounds and the events illustrated in the selected frames.

Optionally, the process may also include a fourth step, in which the audio signal associated with each frame of the video signal is replayed audibly with the associated frames.

Each of these steps will now be described in more detail with reference to the flow diagram shown in Fig. 3. The audio and video signals from the television camera and the microphone are recorded on the video recorder. To analyse the events that took place during the recording, the video recording of the event is replayed and the A/V output signal from the recorder is passed to the computer through A/V input terminal 26. The video card allows the computer to display the television picture on the monitor, for example in a 320x240 bit window. The person operating the computer watches the picture and captures 40 the crucial part of the recording, which in a cricket match may be the moment when the ball passes the bat plus two or three frames before and after that moment. The captured part of the A/V recording can, if necessary, be edited down to three or four frames of greatest interest. This captured part of the A/V signal is stored as a digital AVI file, using a full frame uncompressed format.

WO 00/10333 PCT/GB99/02573 8 The AVI file is split 42 using a computer routine, for example a Visual C++TM routine, to extract the video data, which is saved 44 as an AVI (video only) file, and the audio data, which is saved 46 as a 16-bit text file. The separate video frames making up the video clip are extracted 48 from the AVI (video only) file and saved as a sequence of device independent bitmap (DIB) files, DIBI-DIB,. These files are passed to the data analysis program for graphical display in a GUI environment.

Any time delay between the video signal and the audio signal (caused, for example, by the time taken for the sound to reach the microphone) is compensated for 50 and, optionally, the audio signal may be filtered to remove background noise. The concatenated digital audio signal file is also then converted into a graphical form representing the amplitude of the audio signal against time using the data analysis program. The video and audio data files are organised in matched pairs, each frame of video being matched with a graph of the sound generated during that video frame's construction.

Finally, the selected video frames and the graphical representations of the audio signals associated with those frames are converted 52 into a combined graphical display within the graphical user interface (GUI) environment, for example one running under WINDOWS 95TM This graphical display is converted 54 by the video card into a video output signal.

Optionally, the audio text file may be converted 56 into synchronised audible sound by a sound card, which provides an analogue sound output signal.

Although not shown in Fig. 3, interesting portions ofthe audio signal can optionally be selected for frequency analysis, for example by placing the cursor using a mouse and clicking at the start and end of those sections. The selected signals are analysed by using a fast Fourier transform to generate frequency spectra, which are displayed in the GUI. The entire process may be controlled through the GUI, using standard GUI tools (WINDOWSTM operating environment, icons, mouse and pointer).

An example of an event at a cricket match analysed using the system is shown in Figs. 4a, 4b, 4c and 5. In this example, the batsman was given out bowled, having played the ball onto his wicket.

WO 00/10333 PCT/GB99/02573 9 Figs. 4a, 4b, 4c and 5 represent just one example of a suitable combined graphical display. The display may be presented in many different ways, its main purpose being to illustrate exactly when an audible event took place in relation to the associated visual events. Thus, for example, the display may be as shown in Figs. 4a, 4b, 4c and 5, with a sequence ofthree successive video frames presented side-by-side above a graphical representation of the audio signal associated with those frames. Alternatively, the successive frames may be shown in sequence, the graphical representation of the audio signal associated with each of those frames being shown simultaneously.

In this case, Fig. 4a shows the batsman just before he has played his stroke at the ball, in Fig.

4b he is playing his stroke and in Fig. 4c the ball has passed the bat and he is completing his follow-through. Underneath the three frames, Fig. 5 is a graphical representation of the audio signal associated with those frames. It can be seen that there are three significant audio events (ie three separate sounds), two of these occurring during the second frame (as the ball passes the bat) and the third occurring after the ball has passed the bat. Careful analysis of these events has revealed that the first sound was caused by the ball hitting the edge of the bat, the second by the ball hitting the batsman's thigh pad and the third by the ball hitting the stumps.

Although in this case the batsman was bowled and the umpire was not therefore called upon to make any difficult decisions, the example illustrates the value of the system in analysing exactly what took place. The combined display shows that there were three separate audio events, two ofthese occurring during the second frame as the ball passed the bat. As there was no other obvious source of sound, this suggests very strongly that the ball hit both the bat and the batsman's pads. Therefore, if the ball had missed the stumps and there had been an appeal for the system would have revealed that ball hit the bat before hitting the batsman's pads, and that he should not therefore be given out.

Similarly, if an appeal is made for a catch behind the wicket and the umpire is unsure whether the batsman hit the ball, the system will reveal whether there was a sound at the appropriate moment, as the ball passed the bat. If there was a sound at that moment, then the batsman may have hit the ball. However, if there was no sound, or if a sound was heard but it was found to have been generated before or after the ball passed the bat, then the batsman probably did not hit the ball.

WO 00/10333 PCT/GB99/02573 In a preferred embodiment of the invention, the system includes further refinements for analysing events in even more detail. Such a system includes a computer routine for providing a frequency analysis of selected audio events, which may help to identify the source of those sounds. The frequency analysis is obtained by carrying out a fast Fourier transform on the selected audio events, and the results are then be displayed in graphical form, as shown in Figs.

6 and 7. In this example, Fig.6 represents the frequency analysis of the first audio event (as the ball strikes the edge of the bat) and Fig. 7 represents the frequency analysis of the second audio event (as the ball hits the pads). These sounds have very different frequency spectra, the first sound containing far more high frequency components than the second sound. This is typical of a ball hitting a bat and provides evidence that the ball hit the bat before the pads and not vice versa.

The inventor has discovered that the frequency spectrum of a ball hitting a bat is highly distinctive and identifiable, being substantially independent of how hard the ball hit the bat.

What is more, the spectrum is unlike that of any other sounds likely to be heard at a cricket ground, such as crowd or wind noise, the impact of the ball on the pitch or the pads, the scrape of the bat on the ground, rustling of the batsman's clothing, the fielders' footsteps and so on.

Analysing the frequency spectrum therefore makes it possible to assess whether the bat hit the ball.

Fig. 8 is a block diagram of an alternative system for analysing events, in which a virtual oscilloscope card is provided for generating and displaying a graphical representation of the waveform of the audio signal. A virtual oscilloscope card is a proprietary hardware device that enables a computer to display a trace of a varying-voltage signal, similar to that displayed by an oscilloscope.

The system includes a first computer 60 that includes a video card 62 having an input terminal 64 for the video signal, and a virtual oscilloscope card 66 having an input terminal 68 for receiving the synchronised sound signal. The video clip is displayed in a video window 70 of a combined graphical display 72 and simultaneously the graphical representation of the audio signal associated with the video clip is generated by the virtual oscilloscope card 66 and displayed in an audio window 74. The audio and video signals are synchronised by including a timing correction step to compensate for the time taken by the sound to reach the microphone WO 00/10333 PCT/GB99/02573 11 (typically approx. 0.005s). The combined graphical display is delivered as an A/V output signal through an output terminal 76.

This A/V signal is fed to a second computer 78, which includes a video card 80 that captures the signal and stores it in an AVI file, and allows the separate frames of the signal to be replayed in any sequence as determined by the operator, for detailed analysis of the event depicted. The second computer has a video output terminal 84 through which the video signal representing the graphical display is delivered, for broadcast to the public.

Various modifications of the system are possible. For example, it is not essential to use a video recorder: the live signals from the camera and the microphone can be fed directly into the computer and captured in real time as the ball is delivered. The graphical information can be presented in different formats: for example, the frames of the video signal may be shown consecutively instead of side-by-side. The graphical representation of the audio signal can be broken into sections corresponding to the frames of the video signal, instead of being shown as a single unit. Various other modifications will be apparent to those skilled in the art.

Claims

07-07-2000 GB 009902573 a. 4 4 a* 12 Claims 1. A method of analysing events, in which a television signal of an event to be analysed is recorded, the video part of the recorded signal is stored as a set of data files representing in a graphical format selected frames of the video signal, the audio part of the recorded signal is converted into a graphical format representing the waveform of the audio signal, compensation is provided for any real time delay between the audio part of the recorded signal and the video part of the recorded signal, and the graphical representation of the waveform. of the audio signal is presented with the time- compensated frames of the video signal in a combined graphical format display, allowing the simultaneity of the audio and video events recorded in the television signal to be assessed. 2. A method according to claim 1, in which the video part of the recorded signal is stored as anAVI file. 3. A method according to claim I or claim 2, in which the frames of the video signal are stored as a set of DIB files. 4. A method according to any one of the preceding claims, in which the audio part of the recorded signal is stored as digital data in a text file. Amethod accordn to clai 4, in wic h digital at representing the audio part of the recorded signal is converted into a graphical format by a data analysis program. 6. A method according to any one of claims I to 3, in which the audio part of the recorded signal is converted into a graphical format by a virtual oscilloscope device. 7. A method according to any one of the preceding claims, in which the graphical representation of the audio signal and the selected frames of the video signal are presented in the combined graphical format display within a graphical user environment.
8. A method according to any one of the preceding claims, in which the audio part of the recorded signal is output as an analogue audio output. AMENDED SHEET 07-07-2000 GB 009902573 rage* we s s ea so,* 060 0 0. a0* 13
9. A method according to any one of the preceding claims, in which the audio part of the recorded signal is filtered. A method according to any one of the preceding claims, in which the audio part of the recorded signal is analysed to determine the frequencies present in the audio part.
11. A method according to claim 10, in which the frequency analysis is obtained by means of a fast Fourier transform.
12. A method according to claim 10 or claim 11, in which the frequency analysis is presented graphically in the combined graphical format display.
13. A method according to any one of the preceding claims, in which the combined graphical format display is converted into an A/V output signal.
14. A system for analysing events, said system including a television camera, a microphone, a recording device for recording a television signal of an event captured by the television camera and the microphone, and a data processing device connected to the recording device for analysing the recorded television signal, the data processing device being arranged to store the video part of the recorded signal as a set of data files representing in a graphical format selected frames of the video signal, to convert the audio part of the recorded signal into a graphical format representing the waveform of the audio signal, to compensate for any real time delay between the audio part of the recorded signal and the video part of the recorded signaL, and to present the graphical representation of the waveform of the audio signal with the time-compensated frames of the video signal in a combined graphical format display, allowing the simultaneity of the audio and video events recorded in the television signal to be assessed. A system according to claim 14, in which the video part of the recorded signal is stored as anAVI file.
16. A system according to claim 14 or claim 15, in which the frames of the video signal are stored as a set of DIB files.
17. A system according to any one of claims 14 to 16, in which the audio part of the recorded signal is stored as digital data text file. AMENDED SHEET 14
18. A system according to claim 17, including a data analysis program for converting the digital data representing the audio part of the recorded signal into a graphical format.
19. A system according to any one of claims 14 to 16, including a virtual oscilloscope device for converting the audio part of the recorded signal is converted into a graphical format. A system according to any one of claims 14 to 19, in which the graphical representation of the audio signal and the selected frames of the video signal are presented in the combined graphical format display within a graphical user environment.
21. A system according to any one of claims 14 to 20, in which the audio part of the recorded signal is output as an analogue audio output.
22. A system according to any one of claims 14 to 21, including means for filtering the audio part of the recorded signal.
23. A system according to any one of claims 14 to 22, including means for analysing the audio part of the recorded signal to determine the frequencies present in the audio part.
24. A system according to claim 23, in which the frequency analysis means .••employs a fast Fourier transform process. S 20 25. A system according to claims 23 or claim 24, in which the output of the frequency analysis means is presented graphically in the combined graphical format display.
26. A system according to any one of claims 14 to 25, including means for converting the combined graphical format display into an A/V output signal.
27. A method of analysing events as herein described with reference to the **drawings.
28. A system for analysing events as herein described with reference to the drawings. DATED this twenty-third day of August 2002 o• Allan Plaskett Patent Attorneys for the Applicant: F.B. RICE CO.