AU2012200201B8

AU2012200201B8 - Detecting contacts during sporting activities etc

Info

Publication number: AU2012200201B8
Application number: AU2012200201A
Authority: AU
Inventors: Warren Brennan
Original assignee: Brennan Broadcast Group Pty Ltd
Current assignee: Brennan Broadcast Group Pty Ltd
Priority date: 2006-03-01
Filing date: 2012-01-13
Publication date: 2013-07-18
Anticipated expiration: 2027-03-01
Also published as: AU2012200201B2; AU2012200201A8; AU2012200201A1

Abstract

A method and an apparatus are described for determining parameters of a contact between bodies occurring within an activity, particularly within a sporting, game, s recreational or entertainment activity, such as contact of a cricket ball or baseball with the bat. A digital camera directed towards a target area during the course of the activity captures image data in the infra red region of the spectrum. Means for comparing image data for multiple temporally spaced frames with image data of a reference datum records any localised infra red emission appearing in at least one incident frame, such localised 10 infra red emission being indicative of heat having been generated by a contact which has occurred in the target area and which has generated heat, such as by friction between the bodies or deformation of at least one of the bodies involved in the contact. The image data for at least one of the incident frames is analysed to determine at least one parameter of the contact, and the image data for the incident frame is recorded. Visual display 15 means displays the incident frames including the image data in the IR spectrum by presenting IR data as a visible display of the scene captured for the incident frame so as to enable visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission. 1/+ VISUAL MEANS Fig. I

Description

AUSTRALIA Patents Act 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT ORIGINAL Applicant(s): BRENNAN BROADCAST GROUP PTY LTD Actual Inventor(s): Wan-en Brennan Address for Service: PATENT ATTORNEY SERVICES 26 Ellingworth Parade Box Hill Victoria 3128 Australia Title: DETECTING CONTACTS DURING SPORTING ACTIVITIES ETC. Associated Provisional Applications: No(s).: The following statement is a full description of this invention, including the best method of performing it known to me/us:- DETECTING CONTACTS DURING SPORTING ACTIVITIES ETC Cross reference to related application: This application is a divisional of AU patent application 2007219710 and the entire contents of the specification of that patent application are incorporated herein by this cross reference. 5 Field of the invention: This invention relates to determining the existence and/or a property (such as the location) of a contact such as a contact or an impact in a sporting activity or the like. Background of the invention: The invention relates particularly to sports activities in which a player uses an implement to strike a projectile. For example the to invention can be used in cricket to determine if the ball has contacted the edge of the cricket bat before being caught by a fieldsman and hence to determine objectively if the batsman has been dismissed under the rules of cricket, The invention may also be useable with other sporting, game, recreational, or entertainment activities where parameters of a contact can be relevant to some aspect of the activity, Examples only of such activities 15 and such contacts include: e in the sport of tennis, the nature of contact of the ball with the racket strings, e.g. to impart particular spin to the ball, may be desirably monitored particularly in coaching of a tennis player; - in soccer, hand contact with the ball to block or control its movement is 20 mostly illegal so unambiguous detection of such an impact can be useful; - in baseball and softball whether or not a pitched ball has touched the bat before being caught by the catcher can determine the treatment of the pitch as a strike or as a "foul tip" which means that the batter is out. Although method, apparatus and output products of the invention were originally 25 developed or envisaged, and will be particularly described herein, primarily in relation to determining parameters of the bounce of a tennis ball occurring within a tennis game, it is to be understood that the invention in this divisional application is applicable to other sporting, game, recreational, and entertainment activities involving an implement projectile contact. 30 The determination of the point at which a tennis ball bounces in relation to the line markings or boundaries of the tennis court determines whether the stroke is called legal or is illegal or "out" according to the rules of tennis, Such calling of bounce points is a skilled exercise and in professional tournaments a central umpire and many lines persons located around the tennis court continuously monitor the bounce points to call the shots 35 legal or "out", There is much dissension evident in tennis tournaments concerning the validity of calls made by the umpire and lines persons. To help alleviate such dissension some electronic aids have been proposed or used in the past. These have been based on one of three principles: 2 (1) Impact sensors placed within the court surface so that when a ball strikes the ground, the sensors at that location record the impact and the resulting signals can be monitored to determine objectively whether the bounce point was within or outside the court boundary lines. 5 (2) Light beams have been projected along and transverse to the tennis court in the vicinity of the boundary lines so that when the tennis ball strikes the surface and interrupts the beams, the exact location of the bounce point can be determined from the interrupted beams and hence an objective determination made whether the bounce point is within or outside of the court boundary lines - an apparatus based on this principle is used 10 in some tournaments and is known by the trade mark "Cyclops". (3) Images of the tennis court and surrounds are captured by multiple cameras located at known positions around the court and these cameras continuously capture images including images of the ball in flight which are then analysed to determine trajectories of the tennis ball enabling computation of bounce points by interpolation 15 within the computed trajectories. A simulation of the ball bouncing on the court surface including location of any adjacent court boundary lines can then be displayed for a viewer of the display simulation to determine whether the bounce point was within or outside the court boundary lines. A system based on this general principle has been used in tournaments and is known as the "Hawk Eye" system. 20 These systems can have one or more limitations such as expense of the hardware systems and installations thereof; the difficulty and precision required in providing calibrating or testing and operating of such systems at a tennis venue; limitations on applicability throughout a tennis game since there can be many movements of bodies and many impacts on the court surface during a tennis game that can be registered by such 25 apparatus and, unless accurately discriminated, could lead to false indications - for example the "Cyclops" system is generally used only for determining the bounce point of a tennis player's serve and not subsequent tennis shots. No systems relevant to parameters of contact between an implement and a projectile are known to have been developed and used or to have been proposed in the 30 past. It is an object of the present invention to provide methods and apparatus and output products enabling determining parameters of contact occurring within an activity involving contact between an implement and projectile, Accord ing to a first aspect of the present invention there is provided a method of 35 determining parameters of a contact between bodies occurring within a sporting, game, recreational or entertainment activity which involves a player who uses an implement to strike a projectile, the method including the steps of: 3 directing a digital image capture apparatus, such as a digital camera, towards a target area during the course of the activity and capturing thereby image frames of the target area including image data in the infra red region of the spectrum, said image data in the infra red region of the spectrum including any localised infra red emission appearing 5 in at least one incident frame, such localised infra red emission being indicative of heat having been generated by a contact which has occurred in the target area and which has generated heat by friction between the bodies or by deformation of at least one of the bodies involved in the contact, wherein the target area includes an area where the player is located when using the implement to strike the projectile, and wherein the contact 10 causing localised infra red emission comprises a contact of the projectile with at least one of the following objects within the target area, namely the implement, apparel being worn by the player, accessories being worn by the player, and 15 pails of the body of the player, recording the infra red image data for at least one said incident frame, displaying at least one said incident frame by visual display means by presenting the infra red image data in a visible display of the scene captured for the incident frame so that the displayed image includes a recognisable scene of the player located in the target 20 area with the part of the image depicting the localised infra red emission being superimposed on the recognisable scene so that evidence of the contact is visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission, and enabling determination from the image data for at least one of the incident frames at least 25 one parameter of the contact. Preferably the step of recording the infra red image data for at least one said incident frame comprises recording the infra red image data for several frames preceding the incident frame or preceding each incident frame or preceding the first of a number of successive incident frames, the step of displaying the incident frame(s) being preceded by 30 displaying images from the preceding recorded frames so that a sequence of frames leading up to and including the at least one incident frame is displayed. In the particular preferred method wherein the activity comprises a cricket game and the implement consists of a cricket bat and the projectile consists of a cricket ball; the target area comprises at least part of the cricket pitch where the batsman is stationed; 35 the contact causing localised infra red emission comprises a contact involving the cricket ball, the contact also involving at least one of the following objects within the target area, namely the cricket bat, 4 apparel being wom by the batsman including headgear, clothing, and footwear, accessories being worn by the batsman including protective pads, gloves and helmet, and parts of the body of the batsman; and 5 the method includes a step of analysing the image data to determine at least one parameter of the contact of the cricket ball so as to at least determine the existence of a contact of the cricket ball with at least one of the said objects. . Preferably the method includes the further step of comparing infra red image data for multiple temporally spaced frames with infra red image data of a reference datum so 10 as to detect any localised infra red emission indicative of beat having been generated by a contact by recording any significant increase in the intensity value of the infra red emission within at least one corresponding pixel of successive frames. This significant increase can be detected by searching and analysing the image data for multiple frames to detect the data for a pixel having the highest value infra red is emission of all the frames searched and analysed. Effectively this process involves comparing the image data with image data of a reference datum constituted by the complete set of data from all the frames searched and analysed. Alternatively the step of comparing image data for multiple frames with image data for a reference datum may include comparing the image data for each frame with 20 corresponding image data for preceding frames of captured image data for the same target area so that the image data of the reference datum relates to the target area immediately or shortly before the contact generating the infra red emission. The comparison can then detect and record a substantial increase (eg. beyond a predetermined threshold, which may be an absolute value or a percentage) in the intensity value of infra red emission for a 25 corresponding pixel in successive frames. The step of comparing image data to detect a localised infra red emission may be performed automatically, particularly by execution of an appropriate algorithm by programmed processing means so as to identify the incident frame. The incident frame, or perhaps the frame automatically detected as being the most probable incident frame of 30 interest (e.g. the one having the pixel with the highest value of infra red emission) may then be automatically, or selectively in response to user command, displayed on the visual display means as the relevant captured image of the target area. In one possible embodiment, the process may include the further step of selecting an incident zone within the image of the target area for the displayed incident frame, the incident zone being a 35 selected area within the displayed image in which the localised infra red emission of interest is identified. Preferably the process then includes the further step of further processing the image data for pixels within the selected incident zone to enhance in the image presented by the visual display means the visibility of the infra red emission 5 sources in the captured image. For example, the image data for pixels within the selected incident zone may be processed, e.g. by a digital filtering operation, to enhance the contrast or brightness of pixels having a relatively high infra red emission value compared to nearby pixels. Software for enhancing contrast or brightness is known and commonly 5 provided with picture or image processing software, The selection of the incident zone from within the displayed image may be carried out for example by a user operating an interface device such as a computer mouse to select the area of the displayed image where the localised IR emission of interest is identified by the user inspecting the display. For example, a curser on the displayed image can be positioned at the image of the IR io emission and, upon the user clicking upon that point, the associated software can then automatically enhance the visibility of the IR emission source by enhancing contrast or brightness for the selected pixel and preferably for numerous nearby or contiguous pixels, or for pixels having relatively high IR emission values at, adjacent and/or surrounding the selected point. As an alternative to positioning a curser at a single point, a box may be 15 drawn around the display image area where the localised IR emission of interest is identified, e.g. by positioning a curser and then dragging across the area of interest to create a box constituting the selected area, and the filtering to brighten or increase contrast can be performed only on data for pixels within the box, For example, the filtering can increase contrast of those pixels that have heat values within say 2% of the 20 maximum heat value. Another step of the process that may be performed in a preferred embodiment comprises enhancing the displayed images to enable better user discrimination of features in captured images of the target area. Apart from the enhancement of the IR emissions discussed above, another useful enhancement may comprise image data processing to 25 increase contrast and/or brightness of playing area features such as lines on a tennis court. When a tennis court is in deep shade the IR emission differences between lighter more reflective (and hence cooler) lines and surrounding court surfaces can reduce to make visual discrimination of lines in the displayed image. Upon user selecting e.g. by positioning a cursor on a known image section representing a court line marking, and 30 upon clicking thereon, the processor may filter all data for that image to enhance brightness of all pixels having a substantially identical IR heat value to the selected pixel, say within 2% of that heat value. This will brighten the court making lines in the displayed image. Another process to enhance visibility and/or automatic determination of relevant 35 non-IR emission sources such as tennis court line markings may comprise filtering to enhance image data for a particular pail of the visible spectrum. For example, line markings may incorporate say a yellow or orange reflective component and a filtering process may be performed to increase the brightness count for this pait of the spectrum in 6 the captured images, thereby enhancing visibility of the line markings in a displayed image and/or improving the accuracy of image analysis software to automatically discriminate the line markings. The analysing step preferably includes verification of the or each incident frame 5 as being indicative of a legitimate event in the activity giving rise to a contact of interest by analysing parameters of the infra red emission as recorded in the infra red image data including at least one of: 6 analysing the shape and/or size of the contact area exhibiting emissions in the infra red pail of the spectrum and represented by contiguous pixels of the captured image 10 frame, - analysing an infra red signature indicative of a contact of the type of interest such as the infra red spectral emission characteristics indicative of a pre-determined temperature change, - analysing the duration of decay of the infra red emission in the cluster of 15 contiguous pixels over excessive frames, and - analysing the location of the cluster of contiguous pixels having significant infra red emissions within the target area of the activity. The analysing step may include one or more operations performed manually by an operator of the process, particularly analysing the location of the heat generating contact 20 within the target area - e.g. to consider that location in relation to the rules of the game being monitored. Such analysis may comprise observing the display of the incident frame(s) and determining the location. Also, a step of analysing the shape and/or size of the contact area exhibiting the infra red emissions may be performed by visual inspection of the display and discriminating therefrom whether the heat generating contact is a 25 contact of interest. Such a step of analysing shape and/or size can also be performed automatically by image analysis software to discriminate a heat generating contact of possible interest from one unlikely to be of interest (e.g. an elongated skidding contact of a tennis players' foot compared to a ball bounce). In the preferred embodiment the step of recording the image data for the at least 30 one of the incident frame comprises recording the infra red image data for several frames preceding the incident frame, or preceding each incident frame, or preceding the first of a number of successive incident frames, and the step of displaying the incident frame(s) being preceded by displaying images from the preceding recorded frames so that a sequence of frames leading up to and including the at least one incident frame is 35 displayed to the user. In one possible embodiment, the step of displaying the at least one incident frame by visual display means is carried out in response to user initiation of a display command. 7 The step of analysing the image data to determine at least one parameter of the contact may be used to initiate generation of a signal indicative of a predetermined parameter of the contact. In the case where the activity comprises a cricket game, the method is carried out 5 wherein the target area comprises at least part of the cricket pitch where the batsman is stationed; wherein the contact causing localised infra red emission comprises a contact involving the cricket ball, the contact also involving at least one of the following objects within the target area, namely to the cricket bat, apparel being worn by the batsman including headgear, clothing, and footwear, accessories being worn by the batsman including protective pads, gloves and helmet, and parts of the body of the batsman; and 1s wherein the method includes a step of analysing the image data to determine at least one parameter of the contact of the cricket ball so as to at least determine the existence of a contact of the cricket ball with at least one of the said objects. In a further possible embodiment of the method, the activity comprises any other game which involves a player who uses an implement to strike a projectile, the target area 20 comprises at least an area where the player is stationed when using the implement to strike the projectile, and the contact causing localised infra red emission comprises a contact of the implement with the projectile, and wherein the step of analysing the image data to determine at least one parameter of the contact comprises at least determination of the location on the implement where contact with the projectile occurs. 25 In a second aspect of the invention there is provided a display produced by or resulting from the method according to the first aspect of the invention, the display comprising an image of at least one said incident frame by visual display means in which there is a presentation of the infrared image data in a visible display of the scene captured for the incident frame so that the displayed image includes a recognisable scene of the 30 player located in the target area with the image data relating to the localised infra red emission superimposed on the recognisable scene so that evidence of the contact is visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission, and enabling determination from the image data for at least one of the incident frames at 35 least one parameter of the contact. According to a third aspect of the present invention there is provided apparatus for determining parameters of a contact between bodies occurring within a sporting, game, 8 recreational or entertainment activity which involves a player who uses an implement to strike a projectile, the apparatus including: a digital image capture apparatus, such as a digital camera, directed in use towards a target area during the course of the activity and for capturing thereby image frames of the 5 target area including image data in the infra red region of the spectrum, said image data in the infra red region of the spectrum including any localised infra red emission appearing in at least one incident frame, such localised infra red emission being indicative of heat having been generated by a contact which has occurred in the target area and which has generated heat by friction between the bodies or by deformation of at least one of the 10 bodies involved in the contact, wherein the target area includes an area where the player is located when using the implement to strike the projectile, and wherein the contact causing localised infra red emission comprises a contact of the projectile with at least one of the following objects within the target area, namely the implement, 15 apparel being worn by the player, accessories being worn by the player, and parts of the body of the player, recording means for recording the infra red image data for at least one said incident frame, 20 visual display means for selectively displaying at least one said incident frame including the infra red image data in a visible display of the scene captured for the incident frame so that the displayed image includes a recognisable scene of the player located in the target area with the part of the image depicting the localised infra red emission being superimposed on the recognisable scene so that evidence of the contact is 25 visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission, and enabling determination from the image data for at least one of the incident frames at least one parameter of the contact. The apparatus may further include comparing means for comparing image data for 30 multiple temporally spaced frames with image data of a reference datum so as to detect any localised infra red emission indicative of heat having been generated by a contact by recording any significant increase in the intensity value of the infra red emission within at least one corresponding pixel of successive frames. According to a fourth aspect of the invention there is provided a set of stored data 35 values recording captured image data relating to a sporting, game, recreational or entertainment activity which involves a player who uses an implement to strike a projectile, wherein the stored data values are organised in multiple image frames each of which contains recorded image data captured by digital image capture apparatus directed 9 towards a target area during the course of the activity and the image frames being capable of being sequentially displayed by visual display means so as to present to a viewer of the visual display means a sequence of temporally spaced images of the activity, wherein the stored data values of each image frame record image data in the infra red region of the 5 spectrum, wherein at least one of the image frames comprises an incident frame and the stored data values for the incident frame includes captured infra red image data arising from a localised infra red emission occurring during the activity and which is indicative of heat having been generated by a contact which has occurred in the target area and 10 which has generated heat by friction between the bodies or by deformation of at least one of the bodies involved in the contact, wherein the target area includes an area where the player was located when using the implement to strike the projectile, wherein the contact causing localised infra red emission comprised a contact of 15 the projectile with at least one of the following objects within the target area, namely the implement, apparel being wom by the player, accessories being worn by the player, and paits of the body of the player, 20 and wherein said incident frame when displayed by the visual display means presents the infra red image data in a visible display of the scene captured for the incident frame so that the displayed image includes a recognisable scene of the player located in the target area with the part of the image depicting the localised infra red emission being superimposed on the recognisable scene so that evidence of the contact is visible thereby 25 enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission, and enabling determination from the image data for at least one of the incident frames at least one parameter of the contact. In this fourth aspect, preferably the infra red image data captured and stored in the 30 incident frame and representing the localised infra red emission arising from said contact has been enhanced so as to increase the visibility of the display of the localised infra red emission displayed as evidence of the contact. In the set of stored data values according to the fourth aspect, preferably the recorded infra red data stored in the multiple image frames enables display of a sequence 35 of image frames at and around the incident frame, including displaying a slow motion replay of images of the activity in the target area for multiple frames preceding the heat producing contact and for multiple frames after the contact. 10 In a particular preferred field of use the set of stored data values is created for the activity comprising a cricket game and the implement consists of a cricket bat and the projectile consists of a cricket ball; wherein the target area comprises at least part of the cricket pitch where the 5 batsman was stationed; wherein the contact causing localised infra red emission comprises a contact involving the cricket ball, the contact also having involved at least one of the following objects within the target area, namely the cricket bat, 10 apparel being worn by the batsman including headgear, clothing, and footwear, accessories being worn by the batsman including protective pads, gloves and helmet, and parts of the body of the batsman. Preferably the set of stored data values is arranged so as to be capable of supply 15 for further uses selected from: supply to long term memory for later selective replay for analysis, entertainment, or training; supply to an audience at the location of the activity (sporting venue for example); supply to an umpire or referee for review and judgement; supply to television commentators and audience for discussion and entertainment. In a fifth aspect the invention provides a single data file containing stored data 20 values for an image frame which- contains recorded image data captured by digital image capture apparatus directed towards a target area during the course of a sporting, game, recreational or entertainment activity which involves a player who uses an implement to strike a projectile wherein the data file is a subset of a set of stored data values according to the fourth aspect of the invention, the subset comprising stored data values for a single 25 image frame and wherein the image fi-ame comprises said incident frame. This enables the display and use of a single incident frame showing the contact for identification, verification, and/or analysis, Possible and preferred features of the present invention or technology useful in the present invention will now be described with particular reference to the accompanying 30 drawings. However it is to be understood that the features illustrated in and described with reference to the drawings are not to be construed as limiting on the scope of the invention. In the drawings: Fig 1 is a schematic view of an apparatus according to an embodiment of the present invention. 35 Fig 2 is a flow chart of possible data processing operations in an apparatus of the present invention. Fig 3 is a schematic view of cameras placed to capture images of selected target areas of a tennis court. 11 Fit 4 is a schematic illustration of tennis ball bounces used to illustrate the operation of the present invention. Fig 5 is a schematic illustration of an incident in a cricket game where the present invention can be used. s The apparatus in Fig 1 includes one or more digital cameras 15 directed in use towards a target area such as a sporting, game, recreational, or entertainment area, or area where an activity is to be undertaken. Camera 15 need not be fixed in location but could be movable by an operator, and indeed could be hand held since the process and operations of the invention can be performed provided the camera is directed to the target 10 area when the particular contact incident of interest occurs. The camera could be a digital "still" camera operated to capture an image of the target area shortly after the contact of interest occurs. However for most purposes the implementation of the invention will involve at least one digital video camera which in use is mounted in a known location and is directed towards a respective target area during the course of the activity so as to 15 capture image frames of the target area. A camera capturing fl-ames at 92 frames per second has been successfully tested in a trial of the invention. The camera(s) 15 used in the present invention captures image data in the infra red region of the spectrum. The captured image frames therefore comprise sets of pixels each of which includes image data for infra red emissions detected by the camera arising from within the target area. 20 The infra red data may include spectral data indicative of the temperature of respective locations within the target area giving rise to the infra red emissions (or reflections), or may include infra red emission intensity data, or may include both. If desired however the camera may be constructed or calibrated to record only infra red radiation within a predetermined part of the infra red spectrum selected so as to record infra red emissions 25 from contacts of the kinds of interest (further discussed below) with only the intensity of the detected radiation in that part of the I.R. spectrum being recorded as the infra red data for each image frame. The cameras 15 may also record image data in the visible pail of the spectrum so that for each pixel of each image frame there is not only LR. data recorded but also visible 30 spectrum data recorded. The visible spectrum data may comprise only, or at least, net light intensity in the visible part of the spectrum, such as a grey scale measure or luminance measure, so that by generating an image on a visual display apparatus using the visible spectrum intensity data, a monochrome or grey scale image of the target area is produced for each recorded image frame. This kind of display of course will be familiar 35 to and readily understandable by the user and will enable the user to naturally visualise the target area by viewing the visual display. A suitable digital camera records for each pixel a 14-bit value representing greyscale intensity and IR emission intensity for the respective point of the target area imaged by that pixel. However if desired, each camera 12 may be colour digital video camera capturing colour data such as ROB values for each pixel of each image frame captured (in addition to the IR data). Suitable cameras are made by Cedip of France (www.cedip-infrared.com). The colour data capture may be generally conventional and need not be further described. 5 The image data captured by the cameral 15 is fed to the data processing means 17 which can be a general purpose computer running the required software for image data capture, storage, processing etc. Associated with the data processing means 17 is memory 18 where the image data can be stored and from which the data can be recalled for processing and output to displays and the like. A visual display means 20 associated 10 with the data processor 17 can comprise a conventional visual display unit of a computer or multiple such display units. Other conventional peripherals such as I/O control 21 (eg keyboard, mouse, joy stick, etc.) will be provided for obvious purposes, The display 20 can include a visual display provided for example adjacent the target area where the activity is taking place, eg adjacent a tennis court where an urnpire or referee can view the 15 display unit and in particular can view replayed image frames of the target area to enable viewing of the tennis ball bounce or other contact of interest in the target area. The umpire or referee may have a control unit 21 to instigate replay of image frames captured and stored by the apparatus for repeated viewing and assessment. Likewise the display 20 can be provided in a coaching or training situation for a coach or players to view and 20 review the recorded image frames for learning or other analysis purposes. Another possibility is that the display 20 comprises a large scale visual display provided at a venue of the activity, eg a large scale display provided to an audience viewing a tennis match so that the audience can see a replay of image frames of the target area from a preceding incident in the tennis game. Again another type of display 20 can comprise external 25 displays to which the recorded image frames can be transmitted eg by television transmission or other transmission medium to reach extemal displays such as television sets of an audience located distant from the spoiling venue. Another display 20 preferably used simultaneously in conjunction with any of the preceding displays is a monitor which a technician controlling the process or apparatus views to initiate or 30 modify or terminate certain operations as required, e.g. to initiate a replay, or repeated replays, of displays of captured images. Referring to Fig 2, the functions of the processor 17 can include those illustrated in the flow chart. These processes comprise: (25) Capturing, eg by means of conventional frame grabber means, multiple 35 successive image frames from the outputs of cameras 15, this data including image data in the infra red region of the spectrum and preferably also from the visible spectrum, and storing this data for each frame in memory 18 for later processing. Conventional software can be used for capturing and storing image data, including if desired some preliminary 13 processing such as filtering and perhaps data enhancement such as magnification of IR data, whether universally or when multiple adjacent pixels indicate increased IR emissions relative to a datum such as a mean IR intensity level. The captured data for the multiple successive frames may be stored in memory 18 which may for example include a 5 buffer memory to store a predetermined duration only of activity in the target areas, say 10 to 15 seconds. That is, newly recorded image data will continually overwrite image data from a past time. But this length of time will be sufficient for many uses of the process and apparatus since an operator can selectively stop the capturing process, e.g. by hitting the space bar of a keyboard, and initiate processing and/or viewing and/or io replaying of the image frames including incident frame(s) having a contact of interest in the activity. (26) The image data, particularly the IR data of the pixels of successive captured image frames is compared with a reference datum. The reference datum provides a base enabling detection of any localised, new or transient infra red emission. Therefore the 15 reference datum may for example comprise: (a) corresponding pixels of an immediately preceding image frame, (b) image data from a frame recorded several frames before the subject frame, (c) an average of several preceding frames, (d) a single calibration frame or an average of several calibration frames captured 20 for the target area captured prior to the "live" recording of image frames during the course of the sporting activity, or (e) the set of captured image data for all of the multiple successive frames (from which the highest IR emission level, i.e. the "hottest pixel", from all frames can be detected e.g. by searching all pixels for the pixel data containing the highest JR value, 25 which can be verified as a relevant contact or impact by comparing the IR value of the same pixel with a preceding frame). (27) The detection of any significant IR emission compared to the reference datum is recorded for the respective frame and the frame is noted as an "incident frame" in which there has been a possible or actual localised infra red emission indicative of heat 30 having been generated by a contact which has occurred in the target area and which has generated heat such as by friction between the bodies engaged in the contact (eg the tennis ball contacting the court surface) or by deformation of at least one of the bodies involved in the contact. (28) There may be generated a record or a marker to indicate the relevant frame 35 as an "incident frame", such record or marker comprising for example a data tag associated with the data set for the image frame or perhaps recording in a register other data such as frame number, time of incident, etc. The record or marker is useable to 14 readily identify the incident frame. This is to enable later selective recall and display of the incident frame. (29) The image data for the incident frame is analysed to determine at least one parameter of the contact. 5 (29) (continued) The most basic parameter to determine is whether the incident frame has indeed recorded a contact of the kind of interest, such as a tennis ball bounce. To enable verification of the incident frame as having recorded a legitimate contact of interest, parameters of the recorded infra red emission can be analysed, such parameters including eg (i) the size and shape of the contact as represented by the contiguous cluster 10 of pixels all recording simultaneously the notable increase in the temperature indicative of a frictional contact, the size of the contact needing to be commensurate with the size of a tennis ball (which can be empirically determined by test runs of the apparatus) and the shape of the imprint being substantially as expected, particularly generally elliptical with the long axis in the general direction of travel of the tennis ball. Another parameter of the 15 IR emission enabling verification may be (ii) the infra red signature of the frictional contact, such signature including the change in temperature at the point within the target area being consistent with a frictional contact of interest (again something which can be empirically determined by repeated tests of the apparatus). Another signature may be (iii) the duration of decay of the IR emission - for example it may be determined that the infra 20 red emission from a contact of interest can decay over one or several seconds with a hyperbolic IR intensity decay pattern - this type of verification necessarily entailing analysis of multiple successive image frames in which infra red emission from the same point are detected. (29)(continued) The other parameter of the contact that will be important to 25 analyse in most fields of application of the invention comprises the location of the contact. In particular, in the analysis of the tennis ball bounce, the location of the point of bounce in relation to the court boundary lines can be important. In some embodiments and uses of the invention, the location of the contact point can be determined by the user, eg by an umpire viewing the display 20 when at least one of the incident frames is 30 displayed by the unit 20. That is, the umpire can inspect the displayed incident frame(s) and preceding and succeeding frames if desired, and can analyse and determine the location of the bounce point in relation to the boundary lines of the court. However the invention can also provide automated analysis of the incident frame or frames in determination of the exact location of the contact. For example, the processor 17 may 35 have programmed therein, or stored in look-up tables, or may be able to actively analyse image data captured to determine the spatial relationship of the contact point to other spatial parameters, particularly tennis court lines, For example, the processor may analyse the captured data to discriminate tennis line markings which are provided in 15 contrasting colour to the major part of the tennis court surface such as by using conventional image analysis software to discriminate transitions between areas of different colour or reflectivity (court surface versus marked boundary lines) and to interpolate between discriminated colour transitions in cases where the line markings may 5 be damaged or scuffed or dirty etc. and hence derived mathematically accurate delineations of the court line markings. This "virtual image" of the court line markings can then be superimposed with the derived location of the bounce point and a determination automatically made whether the bounce point falls legally on or within the boundary lines or whether the bounce point is outside the lines. In response to 10 determination of a bounce point falling outside the boundary lines, the system may be operable to generate the signal indicative of that particular parameter of the contact. The signal may be used to initiate the generation of an audible and/or visible alarm in the area where the activity is being perfonned. This can substitute for the call made by a lines person at a tennis tournament. 15 (29) (continued) Another type of analysis of parameters of contact in the sport of cricket entails determining the occurrence of a contact and also for determining with certainty which bodies or articles have been involved in the contact. Examples of contacts which can occur in cricket and their significance include: (i) contact of ball with bat - relevant to determine if a fielder then catching 20 the ball dismisses the batsman, (ii) contact of ball with protective pad on batsman's shin and knee - relevant to determine if a fielder then catching the ball dismisses the batsman, (iii) contact of ball with both bat and pad and in what order or succession 0 relevant to determine if batsman may be dismissed "leg before wicket" (if ball contacted 25 pad first). (iv) contact of ball with batsman's glove - relevant to determine if a fielder then catching the ball dismisses the batsman. (v) contact of ball with the ground just before a fielder catches the ball relevant to determine if the ball has carried to the field to legitimise a 30 dismissal of the batsman. (vi) Contact of the bowler's foot with the pitch near or beyond the line known as the "crease" in bowling the ball towards the batsman - relevant to determine if the bowler has performed an invalid delivery known as a "no ball". 16 (29) (continued) Analogous to these uses for cricket, in baseball and softball a contact incident of interest can be contact of the ball with the batter's bat momentarily before being caught by the catcher (known as a "foul tip"). Objective verification of the contact is possible because of the IR emission from the edge of the bat. 5 (30) The system can, and desirably does, record for those incident frames which have undergone the verification and analysis processes the further results of the analysis and verification so that the incident frames can be flagged and selectively recalled with precision by operators of the system. The storage of incident frames (and preferably ?? immediately preceding and succeeding frames) and if claimed results of analysis and 10 verification such as indicative data flags or markers, may be initiated automatically when such frames have been selectively replayed or initiated by operator intervention such as by an operator inputting a command to store the frame(s) in a more permanent store than the 10-15 second buffer). (31) The system provides for selectively displaying incident frames by the visual is display means 20 to enable visual inspection of the activity in the target area that gave rise to the contact producing the infra red emission. The display will include display of image data in the infra red spectrum which will be presented as a visible display of the scene captured for the incident frame. That is, the IR data will be presented as a visible incident by the display 20 by generating a false colour or false image intensity to be 20 presented by the display. As mentioned above, the cameras 15 preferably record data in the visible part of the spectrum and this visible spectrum data is used to generate on the display a naturally recognisable scene of the target area. With this system the image data relating to the localised infra red emission are used to produce superimposed on the natural scene a contrasting display in the visible part of the spectrum (eg in false colour or 25 enhanced colour or enhanced intensity) so that the contact event is readily visible superimposed on the natural visible scene of the target area. For example, visible spectrum data can be used to produce a monochrome image of activity in the target area readily recognisable to a viewer while the IR data over a threshold can be presented as areas of strongly increased brightness. The display 20 may present under the control of 30 the processor 17 a sequence of several recorded frames preceding and succeeding the start of the detected IR producing contact. In the case of a ball bounce this will mean a display of frames showing the ball moving towards the court surface, bouncing thereon leaving the IR "imprint", and continuing to travel upwardly away from the bounce point. Such a display is not an artificial simulation of the bounce, it is a real image merely also showing 35 the heat imprint of the bounce point enhanced and superimposed so as to be visible on the natural scene. Because line markings on a tennis court surface are generally lighter coloured (more reflective) than the court surface, they may be recorded as darker (i.e. cooler) than the surrounding areas of court surface - therefore to make the display more 17 natural in appearance, the displayed images on the visual display may be "negative" or reversed images, This will make the hot spot caused by a ball bounce to be presented as a darker imprint on the court or line surface. If desired, the control 21 associated with the display 20 can be operated by a user to play or replay the sequence of recorded image 5 frames with the enhanced infra red bounce imprint only on demand, eg in the case of a disputed line call of a lines person at a tennis tournament. Of course such displays of replays of the incident can be made to the audience at the sporting venue, or via television transmissions for enhanced entertainment. A possible complete process performed according to a preferred embodiment of 10 the invention can comprise the following: + Camera captures image frames as 640 by 512 pixel array, each pixel having a 16 bit value containing visible spectrum data and IR data. * Image data, including IR values, are fed to computer. - Store data in short term buffer memory (overwriting oldest data). 15 - Real time visual display of image frames. - User inputs a command to stop capture, e.g. several seconds after the incident of potential interest. * Program searches data in buffer memory to select frame with highest IR value. 20 - Display of image of captured frame having highest recorded IR value, - User inspects displayed image and decides if contact of relevance or interest is present. - If "no" (e.g, if high IR value is result of "flare" such as a solar reflection to the camera from an article in the target area), user can input a command, e.g. striking "N" 25 key of a keyboard, to initiate a further search by the program for the next highest IR value (perhaps limited to a search more than, say, 5 frames away from the first selected frame) and the image frame of which is then displayed. - If "yes", user selects an image point or zone where the contact is identified e.g. by a point and click operation, or by a selection box created by dragging a cursor 30 across the contact zone in the displayed image. - Program processes the image data at or around the selected image point or within the selection box to accentuate visibility of the source of increased IR emissions, e.g. by conventional brightness and/or contrast enhancement software, so as to make the displayed image for that frame (and preferably also the displayed images for immediately 35 preceding and most desirably succeeding frames) clearer for user visualisation of the location and/or nature of the contact. - Enhanced image data for frames at and around the incident frame or first incident frame is displayed. Upon user command, a slow motion replay of the images of 18 the activity in the target area for several frames preceding the heat producing contact and for sufficient frames after the contact is generated on the visual display. . Image signals, particularly after the incident frame has been verified and the IR emissions from the relevant contact enhanced, are supplied for further use - e.g. 5 supplied to long term memory for later selective replay for analysis, entertainment, training, etc, or e.g. supplied to an audience at the location of the activity (sporting venue for example), or e.g. supplied to an umpire or referee for review and judgement, or e.g. supplied to television commentators and audience for discussion and entertainment. From the preceding description the possible use of the invention in tennis as 10 depicted in Figs 3 and 4 can be readily understood. The cameras 15 continuously (or intennittently if desired) capture video images of target areas of the court, the signals from which are processed and recorded. As shown in Fig 4, the bounce of tennis ball 40 on the court surface 41 involves some friction at the bounce point 42. The friction generates heat and hence infra red radiation. The area 43 will have a characteristic size, 15 shape and JR emission characteristics (spectral and temporal) which provide an analysable and therefore verifiable signature of the tennis ball bounce. The location of the bounce point 42 in relation to the court boundary lines 45 following a tennis player's serve will show that the bounce point was beyond the service line and hence the serve was a fault. The bounce point 42a from tennis ball 40a is legal. The bounce point 42b of 20 tennis ball 40b is outside the sideline. Recordings of such tennis ball bounces can be replayed on command at display 20 eg by an umpire for manual assessment and call of the legality, or can be automatically analysed and the legality signalled automatically, In Fig 5 the contact incident of interest comprises contact of the cricket ball 50 with edge of the bat 51 and the point 52. Such a glancing contact with the edge of the bat 25 51 can be difficult for an umpire to detect so that if the ball is caught by the fielder 55 there is uncertainty whether the batsman 53 should be dismissed under the laws of cricket. However with the present invention, the glancing contact at point 52 at the edge of the bat 51 will be a friction event causing an infra red emission detectable using the process and apparatus of the invention. Selective replay by the umpire (who can be off the field of 30 play eg in a studio as is known for higher level televised cricket) can enable ready verification of the existence of the contact 52 with the bat 51 and that off-field umpire can signal accordingly to the on-field officials. It has also been found that there is infra red emission from the part of the ball surface 50a that has made the contact at point 52, so that the captured and displayed images will also reveal evidence of the contact by the 35 brighter IR emission from a part of the ball surface as it is imaged travelling after the initial incident frame. Such evidence can be used in the analysis step of the invention. The use of the invention in baseball and softball is analogous to the described uses in cricket. 19 It will be seen that the process and apparatus and data outputs of the invention provide a useful means for detecting, and in one embodiment automatically analysing and signalling, contacts within sporting and similar activities, such as location of tennis ball bounce points in relation to the line markings and glancing contacts of a cricket ball with 5 a cricket bat, pad, glove, etc.. The system is objective and generates real images of real events, not simulations. 20

Claims

1. A method of determining parameters of a contact between bodies occurring within a sporting, game, recreational or entertainment activity which involves a player who uses an implement to strike a projectile, the method including the steps of: directing a digital image capture apparatus, such as a digital camera, towards a target area during the course of the activity and capturing thereby image frames of the target area including image data in the infra red region of the spectrum, said image data in the infra red region of the spectrum including any localised infra red emission appearing in at least 10 one incident frame, such localised infra red emission being indicative of heat having been generated by a contact which has occurred in the target area and which has generated heat by friction between the bodies or by deformation of at least one of the bodies involved in the contact, wherein the target area includes an area where the player is located when using the implement to strike the projectile, and wherein the contact causing localised 15 infra red emission comprises a contact of the projectile with at least one of the following objects within the target area, namely the implement, apparel being worn by the player, accessories being worn by the player, and 20 parts of the body of the player, recording the infra red image data for at least one said incident frame, displaying at least one said incident frame by visual display means by presenting the infra 25 red image data in a visible display of the scene captured for the incident frame so that the displayed image includes a recognisable scene of the player located in the target area with the pal of the image depicting the localised infra red emission being superimposed on the recognisable scene so that evidence of the contact is visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame 30 and that gave rise to the contact producing the infra red emission, and enabling determination from the image data for at least one of the incident frames at least one parameter of the contact.

2. A method as claimed in claim 1 wherein the step of recording the infra red image 35 data for at least one said incident frame comprises recording the infra red image data for several frames preceding the incident frame or preceding each incident frame or preceding the first of a number of successive incident frames, the step of displaying the incident frame(s) being preceded by displaying images from the preceding recorded 21 frames so that a sequence of frames leading up to and including the at least one incident frame is displayed.

3. A method as claimed in claim 1 or 2 5 wherein the activity comprises a cricket game and the implement consists of a cricket bat and the projectile consists of a cricket ball; wherein the target area comprises at least part of the cricket pitch where the batsman is stationed; wherein the contact causing localised infra red emission comprises a contact involving the 10 cricket ball, the contact also involving at least one of the following objects within the target area, namely the cricket bat, apparel being worn by the batsman including headgear, clothing, and footwear, accessories being worn by the batsman including protective pads, gloves and 15 helmet, and parts of the body of the batsman; and wherein the method includes a step of analysing the image data to determine at least one parameter of the contact of the cricket ball so as to at least determine the existence of a contact of the cricket ball with at least one of the said objects. 20

4. A method as claimed in any one of the preceding claims wherein the method includes the further step of comparing infra red image data for multiple temporally spaced frames with infra red image data of a reference datum so as to detect any localised infra red emission indicative of heat having been generated by a contact by recording any 25 significant increase in the intensity value of the infra red emission within at least one corresponding pixel of successive frames.

5. A method as claimed in claim 4 wherein the step of comparing infra red image data for multiple frames with infra red image data of a reference datum comprises 30 comparing the image data with corresponding image data captured for preceding frames of the same target area so that the image data of the reference datum relates to the target area immediately or shortly before the contact generating the infra red emission.

6. A method as claimed in any one of the preceding claims wherein the step of 35 displaying the at least one said incident frame by visual display means is carried out in response to user initiation of a display command. 22

7. A method as claimed in any one of the preceding claims and including a step of analysing the infra red image data to determine at least one parameter of the contact and in response to the analysis initiating generation of a signal indicative of a predetermined parameter of the contact. 5.

8. A method as claimed in any one of claims 1 to 6 and including a step of analysing the infra red image data to determine at least one parameter of the contact wherein the analysing step includes verification of the or each incident frame as being indicative of a legitimate event in the activity giving rise to a contact of interest by analysing parameters 1o of the infra red emission as recorded in the infra red image data including at least one of: analysing the shape and/or size of the contact area exhibiting contact produced emissions in the infra red part of the spectrum and represented by contiguous pixels of the captured image frame, . analysing an infra red signature indicative of a contact of the type of 15 interest such as the infra red spectral emission characteristics indicative of a pre determined temperature change, a analysing the duration of decay of the infra red emission in the cluster of contiguous pixels, and - analysing the location of the cluster of contiguous pixels having significant 20 infra red emissions within the target area of the activity.

9. A display produced by or resulting from the method as claimed in any one of the preceding claims, the display comprising an image of at least one said incident frame by visual display means in which there is a presentation of the infra red image data in a 25 visible display of the scene captured for the incident frame so that the displayed image includes a recognisable scene of the player located in the target area with the image data relating to the localised infra red emission superimposed on the recognisable scene so that evidence of the contact is visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact 30 producing the infra red emission, and enabling determination from the image data for at least one of the incident frames at least one parameter of the contact.

10. Apparatus for determining parameters of a contact between bodies occurring within a sporting, game, recreational or entertainment activity which involves a player 35 who uses an implement to strike a projectile, the apparatus including: a digital image capture apparatus, such as a digital camera, directed in use towards a target area during the course of the activity and for capturing thereby image frames of the 23 target area including image data in the infra red region of the spectrum, said image data in the infra red region of the spectrum including any localised infra red emission appearing in at least one incident frame, such localised infra red emission being indicative of heat having been generated by a contact which has occurred in the target area and which has 5 generated heat by friction between the bodies or by deformation of at least one of the bodies involved in the contact, wherein the target area includes an area where the player is located when using the implement to strike the projectile, and wherein the contact causing localised infra red emission comprises a contact of the projectile with at least one of the following objects within the target area, namely 10 the implement, apparel being worn by the player, accessories being worn by the player, and parts of the body of the player, 15 recording means for recording the infra red image data for at least one said incident frame, visual display means for selectively displaying at least one said incident frame including the infra red image data in a visible display of the scene captured for the incident frame so 20 that the displayed image includes a recognisable scene of the player located in the target area with the part of the image depicting the localised infra red emission being superimposed on the recognisable scene so that evidence of the contact is visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission, and 25 enabling determination from the image data for at least one of the incident frames at least one parameter of the contact.

11, Apparatus as claimed in claim 10 and fluther including comparing means for comparing image data for multiple temporally spaced frames with image data of a 30 reference datum so as to detect any localised infra red emission indicative of heat having been generated by a contact by recording any significant increase in the intensity value of the infra red emission within at least one corresponding pixel of successive frames.

12. A set of stored data values recording captured image data relating to a sporting, 35 game, recreational or entertainment activity which involves a player who uses an implement to strike a projectile, wherein the stored data values are organised in multiple image frames each of which contains recorded image data captured by digital image capture apparatus directed towards a target area during the course of the activity and the 24 image frames being capable of being sequentially displayed by visual display means so as to present to a viewer of the visual display means a sequence of temporally spaced images of the activity, wherein the stored data values of each image frame record image data in the infra red region of the spectrum, 5 wherein at least one of the image frames comprises an incident frame and the stored data values for the incident frame includes captured infra red image data arising from a localised infra red emission occurring during the activity and which is indicative of heat having been generated by a contact which has occurred in the target area and which has 10 generated heat by friction between the bodies or by deformation of at least one of the bodies involved in the contact, wherein the target area includes an area where the player was located when using the implement to strike the projectile, 15 wherein the contact causing localised infra red emission comprised a contact of the projectile with at least one of the following objects within the target area, namely the implement, apparel being worn by the player, 20 accessories being worn by the player, and parts of the body of the player, and wherein said incident frame when displayed by the visual display means presents the infra red image data in a visible display of the scene captured for the incident frame so 25 that the displayed image includes a recognisable scene of the player located in the target area with the pait of the image depicting the localised infra red emission being superimposed on the recognisable scene so that evidence of the contact is visible thereby enabling visual inspection of the activity in the target area at the time of the at least one incident frame and that gave rise to the contact producing the infra red emission, and 30 enabling determination from the image data for at least one of the incident frames at least one parameter of the contact.

13. A set of stored data values as claimed in claim 12 wherein the infra red image data captured and stored in the incident frame and representing the localized infra red emission 35 arising from said contact has been enhanced so as to increase the visibility of the display of the localised infra red emission displayed as evidence of the contact. 25

14. A set of stored data values as claimed in claim 12 or 13 wherein the recorded infra red data stored in the multiple image frames enables display of a sequence of image frames at and around the incident frame, including displaying a slow motion replay of images of the activity in the target area for multiple frames preceding the heat producing 5 contact and for multiple frames after the contact.

15. A set of stored data values as claimed in any one of claims 12 to 14 wherein the activity comprised a cricket game and the implement consisted of a cricket bat and the projectile consisted of a cricket ball; 10 wherein the target area comprised at least part of the cricket pitch where the batsman was stationed; wherein the contact causing localised infra red emission comprises a contact involving the 15 cricket ball, the contact also having involved at least one of the following objects within the target area, namely the cricket bat, apparel being worn by the batsman including headgear, clothing, and footwear, accessories being worn by the batsman including protective pads, gloves and 20 helmet, and parts of the body of the batsman.

16. A set of stored data values as claimed in any one of claims 12 to 15 wherein the set of stored data values are arranged so as to be capable of supply for further uses 25 selected from: supply to long term memory for later selective replay for analysis, entertainment, or training; supply to an audience at the location of the activity (sporting venue for example); supply to an umpire or referee for review and judgement; supply to television commentators and audience for discussion and entertainment. 30

17, A data file containing stored data values for an image frame which contains recorded image data captured by digital image capture apparatus directed towards a target area during the course of a sporting, game, recreational or entertainment activity which involves a player who uses an implement to strike a projectile wherein the data file is a subset of a set of stored data values as claimed in any one of claims 12 to 16, the subset 35 comprising stored data values for a single image frame and wherein the image frame comprises said incident frame. 26