GB2488991A

GB2488991A - A system for measuring audio signals generated during sporting activity

Info

Publication number: GB2488991A
Application number: GB1104063.1A
Authority: GB
Inventors: Andrew Mason; Graham Thomas; David Marston; Chris Pike; Chris Baume; Michael Evans; Michael Armstrong; Sam Davies; Andrew Gibb; Anthony Churnside
Original assignee: British Broadcasting Corp
Current assignee: British Broadcasting Corp
Priority date: 2011-03-09
Filing date: 2011-03-09
Publication date: 2012-09-19
Also published as: GB201104063D0

Abstract

A system for producing an output signal representative of characteristics of a sporting action from an audio signal received from a sporting event comprising an input 10 for receiving an audio signal, a processor 14 to derive features from the signal, an analyser to analyse the features of the signal with reference to the characteristics of the equipment that produced the signal and an output device 20. The system may analyse the frequency, amplitude, cepstral coefficients or prosodic parameters of the audio signal and the output 20 may provide a graphical representation. Also disclosed is a system which compares audio signals with those in a database to identify the closest match. A further disclosure provides for a system which analyses an audio signal at a range of spaced sensors to make calculations in respect of the direction of the sound which may include the use of spherical harmonic decomposition.

Description

S

METHOD AND APPARATUS FOR SPORTS AUDIO ANALYSIS

BACKGROUND OF THE INVENTION

This invention relates to methods and apparatus for analysis of audio from a sporting event. In particular, the invention relates to providing an output based on sound emitted from an activity within a sporting event.

A given activity or action within a sporting event may emit a sound. For example, a ball hitting a tennis racket or cricket bat will emit a sound. The current best known apparatus for analysis of a sound from a sporting action is the so called "snickometer" used in cricket to assist in determining whether a cricket ball and bat have made contact. The snickometer operates by producing a graphical representation of the audio wave form of sound received by a microphone in the vicinity of the batsman and displaying this in synchronisation with television images of the cricket match. This can assist a human in interpreting whether or not a bat and ball have made contact.

In the sport of tennis, a microphone is used to receive audio emitted in the vicinity of the tennis net to assist in deciding whether or not a tennis ball has hit the top of the net. As with the example of a cricket bat and ball, this analysis is simply the presence or absence of the sound of contact between the ball and another entity.

SUMMARY OF THE INVENTION

We have appreciated limitations of prior devices used in sports audio analysis. In particular, we have appreciated that improvements may be made both to the processing of audio signals to derive a measure of the quality of an action with a sporting event and also that improvements may be made in the output by which the quality of the sound related to an action may be provided to control a devoce related to the sporting event or provided to a user device for presentation as an audio or video indication.

The invention may be summarised in three broad aspects.

In a first aspect, a system is provided comprising an input for receiving an audio signal, a processor for deriving features from the input signal, an analyser for analysing the features with reference to characteristics of the sporting equipment that produced the sound and an output for asserting an output signal such as a signal to an audio or video device. A device according to the first aspect may, for example, analyse particular amplitude or frequency features from sound emitted from sporting equipment and, using parameters from the sporting equipment, determine features of the action that cause the emitted sound.

In a second aspect of the invention, a system comprises an input for receiving audio from an action within a sporting event, a processor for extracting features of the audio signal, a database holding previously stored features of audio signals, each relating to a particular type of sporting action, a comparator for comparing the features of the received audio signal against features in the database to determine the closest match and an output for asserting an output signal such as an audio or video signal to a device for presentation of the closest match. A device operating according to this aspect may thereby determine which of various pre-stored actions is the best match for the current action that has occurred.

In a third aspect of the invention, a system comprises an input for receiving audio signals from multiple geographically spaced sources at a sporting event, a processor for deriving at least two features of the received audio, one of the features relating to directionality of sound, and an output for providing a signal for assertion to a video device to provide a graphical representation of the directionality characteristic and at least one other feature in conjunction with a video signal from the sporting event.

The three aspects of the invention may be used in conjunction with one another. S 3

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described in more detail by way of example with reference to the drawings, in which: Figure 1 is a diagram of a system which may embody the three aspects of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the invention in its various aspects is shown in Figure 1. The apparatus could be embodied in a self-contained device, but preferably comprises various components which are distributed and together form a sports event system for processing audio signals received from the sounds generated by actions within sporting events.

Microphones 10 are positioned at a sporting event so as to detect audio from actions such as ball hilling bat or racket, footfalls on the ground, cyclists passing the microphone and the like. Other audio receivers are possible, including hydrophones, transducers and other devices capable of converting sound to electrical signals. In addition, the microphone may be part of a television camera and provide the audio signal along with the video signal.

In use, the preferred embodiment will have multiple microphones 10 arranged around the source of audio or distributed within an arena so as to provide information on directionality of sound generated by actions within the sporting event.

The microphones 10 are coupled to a converter 12 for providing digital signals, here shown as an analogue to digital converter. The converter provides the signals to a processor 14 here shown as a CPU for processing the audio signals.

The processor may be a dedicated circuit or a general purpose processor. A video signal 22 is received by a video processor 16. The video processor 16 provides the video in an appropriate format for combining with the signal asserted from CPU 14 on line 23 in a combiner shown as mixer 18 for providing an output, S 4 typically as combined audio video data to an output device such as a display 20.

In this way, audio representative of actions within a sporting event may be combined with the video received on line 22 and asserted in conjunction with the video after processing according to processing steps in accordance with aspects of the invention.

The system also includes a database 24 which holds data representing features of previously recorded actions within sporting events or data relating to characteristics of sporting equipment by which an acoustic response may be derived. The operation of the CPU 14 in conjunction with the database 24 will now be described in greater detail.

The processor 14 receives the digitised audio signals, whether from microphones or via extraction from an audio video feed, and processes the signals to isolate a temporal portion of the signal relating to the sporting action. For example, the sound of a ball on bat, racket or hilling a surface occurs for a defined short time period and this portion of the signal is isolated for further analysis. The processing techniques may include using cepstral coefficients to analyse the timbre of the sound-produced, speech-model analysis to deduce the prosidy of in-game utterances (for example grunts) and other such techniques.

Subsequently, the portion of the signal relating to an action is further processed by the processor 14 to render the signal in an appropriate form for comparison to prior stored data representing previously recorded audio signals of the same type of action. A variety of comparison algorithms are possible, but the preferred approach is to process the audio signal to produce one or more of data representative of the frequency components of the signal, relative amplitude with respect to time, the audio features described above such as cepstral coefficients, speech model outputs, and so on, and to compare this using a standard comparison algorithm. Such algorithms include neural networks, hidden Markov models, Bayesian statistical analysis, genetic algorithms or other such algorithms. The features of the audio signals can be processed using support vector machines and k-mean clustering to group auditory events into semantic clusters. As an example "a good serve" could be a semantic cluster. S 5

Using a best fit algorithm, the data from a previously recorded similar sporting action that matches most closely to the audio being processed can be retrieved.

In the example of a bat and ball, for example the sound of the ball hitting a tennis racket might reveal where on the racket the ball may have contact, the speed and acceleration of the ball, spin and other such data relating to the nature of the action. This information may be provided by finding the corresponding action that closely matches within the database and retrieving from the metadata stored with the audio representing that action the matters described such as position on the racket, speed and so on. The parameters relating to the action currently performed may thereby be inferred based on the similarity to previously recorded actions.

A second approach to using the processor and database relates to storing parameters of the sporting equipment or environment and using those parameters as part of the comparison process. Accordingly, instead of comparing the processed features of the currently recorded audio against previously stored recordings, features may be compared against data derived from the parameters of the equipment and surroundings. As an example, a cricket bat has a given length and density, whereas a cricket glove has a rather different, irregular shape and density. Using known techniques, such as finite element analysis, taking account of physical parameters such as mass, dimensions, elasticity, damping, the resonant frequencies, and sound-producing mechanisms of the objects may be derived and used in the comparison against the frequencies found in the audio segment relating to the recorded action.

This approach may have certain advantages in contrast to comparing to a database of pre-recorded sounds. In sporting activities involving a sports arena, the dimensions and composition of the arena may contribute to the nature of the sounds as well as the dimensions and composition of the sporting equipment. By providing a set of calculations to determine the likely frequencies and resonances of actions within the sporting activity, a better comparison may be made than can be achieved by comparing two pre-recorded characteristics. This is because the pre-recorded characteristics may vary considerably based on the precise make, composition and size of the sporting equipment and room or arena in which the sound is recorded.

The analysis may be further refined by applying a psycho-acoustic model to the input or model output to approximate how the physical sounds would be perceived by a listener. This could, for example, be used to indicate that a sound would not have been heard by an official.

The result of the comparison is an output signal which is asserted on line 23.

The output signal may be used to control further devices in a variety of ways. For example, the signal may control equipment within the live sporting event itself, such as controlling a barrier if the audio processing reveals that a sporting activity has commenced before the start gun, controlling a red light to alert participants that some sort of infringement of rules has occurred, such as a foot placed outside specified locations of a sporting track or pitch. A particular benefit, though, is in providing a visual indication to an umpire or other viewer. In this regard, there is synergy between the audio and video data. For example, the video data may be used to determine when an audio signal should be analysed.

The video data may, for example, be used to determine the position of a given action being viewed and then the audio relating to the video frame showing that action can be processed as described above. Furthermore, the visual indication of some characteristic of the audio action may be mixed and overlaid on the video in a mixer 18 as will now be described.

A very simple manner of mixing may be to overlay a traffic light or a similar visual indication to give an indication of the overall quality of a given sporting action.

For example, green to indicate a ball did not hit a bat or red to indicate that the balI did hit the bat.

A more complex visual indication is preferred, though, according to a third aspect of the invention by using data relating to the directionality of audio as perceived by microphones 101 in addition to at least one other characteristic such as frequency or amplitude. To provide the directIonality of the sound, preferably multiple microphones 10 may be arranged around the sporting arena or at least around the area of play that results in the action for which the audio is to be processed. By comparing audio signals received from each microphone and time of arrival, as well as relative amplitude, information relating to the position of occurrence of the action and directionality of the sound may be produced and S 7 asserted on the signal line 23. Position and directionality information may also be represented by a spherical harmonic decomposition, or a binaural representation of the audio signals. The mixer may then produce a graphical representation for overlay on the video. For example, in the case of a sporting arena, the relative amplitude of the sound produced in different directions from a central point may be converted to a visual display showing relative amplitude by thickness of a displayed line and relative frequency by the colour of the displayed line. A particular example of this may be to provide an overlay surrounding a batman's bat or a tennis player's racket showing the directionality of the sound and relative amplitudes or frequency.

By way of further example, various sporting activities will now be described and the use of the apparatus embodying any of the three aspects of the invention exemplified.

Cricket The sound of a ball and a cricket bat typically falls in a particular range of frequencies and amplitudes. In order to better discriminate between different types of shot and, indeed, whether the ball hit the bat, the frequencies may be processed to assert an output signal that provides a greater discrimination between frequencies and thereby give a visual display of the sound. For example, a ball hitting the edge of the bat may give a higher frequency sound and by allocating this frequency to the colour red in comparison to the frequency from a shot in the centre of the bat, which may be blue, a clear visual distinction may be given as to where on the bat the ball struck.

Tennis Discrimination may be made as to whether a ball hit the top band of a net or not by knowing the length and tension of the top band and thereby the frequencies that will be given. The sound coming from the top band may thereby be discriminated from other sounds. This would not require prior recording of sounds but deriving the comparison from physical attributes of the tennis net. S 8

Running The foot falls of runners passing an array of microphones may provide a characteristic related to stride, length, weight or other effort on the part of a runner.

Diving The entry of a diver into water should be clean with the minimum of splash. By reviewing the profile of a diver as they enter the water and comparing this to the audio sound, a database of clean verses less clean dives may be created as well as dives of different angles of entry into the water. By comparing the timing, frequency and amplitude of the recorded sound against the prestored database, an automatic indication of a characteristic of the entry into the water may be given.

Rowing The synchronisation of the oars is of importance to the rowing performance.

Using a hydrophone or sensors on the oars audio signals may be produced from which the stroke rate, timing and other characteristics may be derived and asserted either to give direct feedback to the sportsman or to give a visual indication. Golf

A golf club hilling a ball will produce a different sound depending upon the position on the face of the gold club and will have directionality depending upon the direction of the ball. By surrounding the tee with an array of microphones, this directionality and difference in frequency may be derived to give a visual indication of the nature of the stroke. S 9

Claims

CLAIMS1. A system for producing an output signal representative of characteristics of a sporting action from an audio signal received from a sporting event, comprising: an input for receiving an audio signal, a processor configured to derive features from the input audio signal, an analyser configured to analyse the features with reference to characteristics of the sporting equipment that produced the audio signal to produce an output signal: and an output for asserting the output signal to an output device.
2. A system according to claim 1, wherein the features comprise frequency components.
3. A system according to claims I or 2, wherein the features comprise time varying amplitude.
4. A system according to any preceding claim wherein the features comprise cepstral coefficients.
5. A system according to any preceding claim wherein the features comprise prosodic parameters
6. A system according to any preceding claim, wherein the characteristics of the sporting equipment include the length of at least one dimension providing vibrations on occurrence of the sporting action.
7. A system according to any preceding claim, wherein the analyser is configured to use a model based on properties including one or more of mass, dimensions, elasticity or damping relating to the sporting equipment. S 108. A system according to any preceding claim, wherein the output device comprises an apparatus for interacting with the sporting event.9. A system according to any preceding claim, wherein the output device comprises a display.10. A system according to any preceding claim, wherein the output device comprises a mixer for mixing a graphical representation with a video signal from the sporting event.11. A system according to any preceding claim, wherein the analyser is configured to receive a video input from the sporting event and to determine a time window for analysing the features based on the video input.12. An audio video system comprising the system of any preceding claim and an output device comprising a mixer for mixing a graphical representation with a video signal from the sporting event.13. A system for producing an output signal representative of characteristics of a sporting action from an audio signal received from a sporting event, comprising: an input for receiving an audio signal, a processor for extracting features of the audio signal, a database holding previously stored features of audio signals, each relating to a particular type of sporting action, a comparator for comparing the features of the received audio signal against features in the database to determine the closest match to produce an output signal and an output for asserting the output signal to an output device.14. A system according to claim 13, wherein the features comprise frequency components.15. A system according to claims 13 or 141 wherein the features comprise time varying amplitude. S 1116. A system according to any of claims 13 to 15, wherein the previously stored features of audio signals are classified by machine inference algorithms 17. A system according to any of claims 13 to 16, wherein the previously stored features of audio signals are derived from recordings of sporting actions.18. A system according to any of claims 13 to 17, wherein the output device comprises an apparatus for interacting with the sporting event.19. A system according to any of claims 13 to 18, wherein the output device comprises a display.20. A system according any of claims 13 to 19, wherein the output device comprises a mixer for mixing a graphical representation with a video signal from the sporting event.21. A system according to any of claims 13 to 20, wherein the analyser is configured to receive a video input from the sporting event and to determine a time window for analysing the features based on the video input.22. An audio video system comprising the system of any of claims 13 to 21 and an output device comprising a mixer for mixing a graphical representation with a video signal from the sporting event.23. A system for producing an output signal representative of characteristics of a sporting action from an audio signal received from a sporting event, comprising: an input arranged to receive an audio signal from multiple geographically spaced sources at a sporting event, a processor for extracting at least two features of the audio signal, one of the features relating to directionality of sound, and an output for providing a signal for assertion to a video device to provide a graphical representation of the directionality feature and at least one other feature in conjunction with a video signal from the sporting event. S 1224. A system according to claim 23, wherein the features comprise frequency components.25. A system according to claims 23 or 24, wherein the features comprise time varying amplitude.26. A system according to any of claims 23 to 25, wherein spatial characteristics of the sound are represented by a spherical harmonic decomposition.27. A system according to any of claims 23 to 25, wherein spatial characteristics of the sound are represented binaurally.28. A system according to any preceding claim, wherein the features comprise time-delay and amplitude differences between the signals from the elements in an array of sensors 29. A system according to any preceding claim, wherein the output device comprises a mixer for mixing a graphical representation with a video signal from the sporting event.30. A system according to any preceding claim, wherein the analyser is configured to receive a video input from the sporting event and to determine a time window for analysing the features based on the video input.31. A system according to any preceding claim, wherein in the features are further processed by a psycho-acoustic model.