AUSTRALIA Patents Act 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT ORIGINAL Applicant(s): BRENNAN BROADCAST GROUP PTY LTD Actual Inventor(s): Warren Brennan Address for Service: PATENT ATTORNEY SERVICES 26 Ellingworth Parade Box Hill Victoria 3128 Australia Title: PROCESSING AND PRESENTING THERMOGRAPHIC IMAGES Associated Provisional Applications: No(s).: 2008905360 The following statement is a full description of this invention, including the best method of performing it known to me/us:- PROCESSING AND PRESENTING THERMOGRAPHTC IMAGES This invention relates to the processing and presenting of thennographic images particularly, although not exclusively, for presenting information for education, entertainment, or amusement purposes. 5 It has been proposed in the past to utilise cameras having sensitivity in the infrared region of the electromagnetic spectrum to capture images of subjects of interest and process the images to present information concerning infrared emissions in a false colour representation. For example, this technology can be used in testing of products, such as manufactured articles including electronic circuits so as to detect infrared 10 emissions that might be indicative of normal functioning or, alternatively, indicative of a fault condition. Effectively the thermographic images are being used for testing or analysis or diagnostic purposes. Another field is in satellite imaging of the earth's surface so as to collect and present useful information or data about vegetation, photosynthetic activity, or human produced phenomena. These fields of use tend to be 15 static, sometimes still image frames, collected, analysed and presented or displayed, or perhaps being used to trigger industrial processes. Although these prior systems may have been providing information or serving educational purposes or functions, such prior thermographic data analysis and presentation have not been utilised for education, entertainment or amusement of nembers of the general public but rather have tended to 20 be specialist technical or scientific tools. It is an object of the present invention to provide methods and systems for processing and presenting thermographic images in a manner that can provide wide or general appeal or interest, e.g. for information, education, entertainment or amusement purposes, 2 According to one aspect of the present invention, there is provided a method of processing and presenting thernographic images including the steps of: providing a digital camera having both infrared and visible spectrum sensitivity, locating and directing the camera at a subject field in which dynamic activities of target 5 subjects occur, and generating by the camera digital image data which comprises data for at least one image frame and which further comprises for each pixel of the or each image frame both infrared and visible spectrum data; inputting the image data. for the pixels of the or each image frame to a processing system; 10 providing a thermal input to the processing system, the thermal input comprising a specification or selection of isotherm parameters which comprise an infrared or temperature value or a range of infrared or temperature values; operating the processing system to identify pixels of the or each image frame having infrared data representing inFrared or temperature values of the target subjects 15 satisfying the specified or selected isotheri parameters so as to thereby create a selected pixel set; applying a visibility enhancing false colour value to image data of each pixel in the selected pixel set, and visibly displaying the or each image frame so that the pixels of the selected pixel 20 set representing elements of the view of the subject field captured by the camera exhibiting infrared emissions satisfying the isotherm parameters are visibly displayed in an enhanced or false colour superimposed on a visible spectrum image composed of pixels of the image frame other than those in the selected pixel set. 3 By providing an input comprising a specification or selection of isotherm parameters an operator can select not only a particular temperature on a scale but can also select upper and lower limits of temperature values so as to optimise the information, entertainment or amusement generated by the presented image. For example only, if the 5 target subjects within the subject field comprise mammals, such as racehorses, selecting isotherm parameters indicative of a temperature value or range of values a little above normal body temperature of the mammal, can produce display images in which heat generation by the target subject, and more particularly by particular areas of the body of the target subject can be generated and visually presented to indicate a particular level of 10 temperature increase. For example, a racehorse during a horse race may be found to be exhibiting significant infrared emissions indicative of body temperature, or at least particular muscular temperature, significantly above the normal body temperature of the animal as the end of the horse race is nearing. By selecting temperature values, say a few degrees above normal body temperature, either the entire animal or, alterniatively, 15 particular parts of the body of the animal, may exhibit significant infrared emission as the animal nears the end of the horse race. This may be a normal temperature increase due to the strenuous exercise, or may be indicative of the animal having worked excessively hard before the end of the horse race is reached, or may be indicative of some other noteworthy cause, such as anxiety, inflammation, or infection of the racehorse in which 20 case the relatively high infrared emissions may be discriminated and displayed early in a horse race or perhaps even before the horse race commences. The method can thus provide information or entertainment, or amusement, e.g. by being presented in association with commentary by sports commentators in the broadcast or other presentation of the horseracing event. 4 In. a preferred implementation, the target subjects comprise mammals engaged in the dynamic activities, and the step of providing the thermal input comprising specifying or selecting isotherm parameters indicative of a temperature value or range of values a little above normal surface body temperature of the mammals whereby the step of visibly 5 displaying the or each image frame produces displayed images in which particular levels of increase of surface temperature of the mammals' bodies are visually presented. The mammals may comprise racehorses whereby the method produces visible displays of significant infrared emissions from the racehorses' body surfaces indicative of body temperature, or at least indicative of particular muscular temperature, significantly above 10 the normal body temperature of the racehorses. In this embodiment, the method further includes locating and directing the camera so that the image field of view comprises a race track along which the racehorses travel, the portion of the race track comprising at least the final stages leading towards and eventually reaching the line at the end of a horse racing event. 15 Preferably the processing system provides an interface for an operator to utilise in performing the step of providing the thermal input, the interface of the processing system including a display with which the operator can interact to specify or select the isotherm parameters by a process selected from: - pointing to and clicking on an IR or temperature value displayed, 20 - touching the display screen on which a scale of IR or temperature values is presented to thereby select the desired IR or temperature value, and - pointing to and clicking on, or touching the display at, a part of the displayed image so that the IR value of a pixel or body of pixels at that part of the displayed image becomes the central IR or temperature value or indicates a range of TR or temperature 25 values surrounding the indicated pixels IR value is automatically selected. 5 Instead of entirely manual generation or selection of the thermal input, the step of providing the thermal input may comprise programming and operating the system to present to the operator an image of a. target subject within the subject field, and then specifying or selecting by means of the operator indicating a single point or a general 5 area of the displayed image, and then automatically reading or analysing the particular IR value or values at that selected part of the displayed image as constituting the selected isotherm parameter. In one embodiment, the steps of the method are performed in real time by continuously generating image frames by the camera which are displayed to the operator, and under 10 real time control of the operator selecting isotherm parameters while viewing the sequence of displayed image frames, processing the digital image data and presenting on a display images in which infrared emissions from target subjects within the images being captured by the camera would satisfy the selected isotherm parameters so they are visibly displayed in an enhanced or false colour image superimposed on visible spectrum 1I image of the dynamic activities ofthe target subjects. In an alternative embodiment, the visible display of the image frames with enhanced or false colour displays of pixels satisfying the isothern parameters is performed as a replay of the dynamic activities of the target subjects, the method comprising the steps of generating the image data for multiple frames, feeding the image data to a. storage or 20 memory associated with the processing system, and then subsequently presenting the image data. to the operator, who carries out the step of providing the thermal input, including iteratively carrying out that step to optimise the final visible display by testing different selected isotherm parameters so as to produce an optimised and most informative or entertaining image display, and finally presenting the optinised display as 25 a replay of the dynamic activities. 6 Preferably the method further includes the step of frame image data processing to discriminate the dynamic activities of target subjects as distinct from extraneous, superfluous, background or transient sources or events producing IR emissions- The discriminating step may include defining within the or each image frame a limited area 5 within which the selected pixel set is to be confined, whereby only IR data satisfying the isotherm parameters within that limited area is enhanced and displayed in the false colour, while IR emissions, even if satisfying the isotherm parameters, fi-om areas outside the defined frame parts will not be enhanced and displayed in the false colour. In one possible system, the step of defining the limited area may be under operator 10 control, and may comprise the operator viewing one or a sequence of image frames and defining the frame parts of interest within the or each frame by either: - pointing and clicking at a part of the image frame and the system operates so that only pixels within a predetermined radius of that central pixel are analysed, or * clicking and dragging to create a frame or box in each image within which the 15 image data analysis is performed to create the selected pixel set for IR colour enhancement. In an alternative possible system, the step of defining the limited area is performed at least partially automatically by capturing images of the subject field shortly before the dynamic activities of interest occur, storing these captured background images and 20 comparing them to the corresponding captured image frames during the course of the dynamic activities of the target subjects, whereby processing the captured image frames of interest so as to compare them with stored corresponding image frames, the dynamic activities of the target subjects are discriminated against background and only pixels representing captured image data for the target subjects are then be analysed to identify 25 the pixels constituting the selected pixel set. 7 The processing system may perform a step of discriminating target subjects and confining the processing of image data to image frame data for the discriminated target subjects and eliminating from further processing image frame data from areas of the captured image frames other than those areas discriminated as relating to target subjects 5 of interest, the discriminating of target subjects being performed by image frame data processing so as to: identify bodies changing relative position within successive captured image frames so as to thereby identify moving target subjects and discriminating them from background or other objects not constituting target subjects of interest, or 10 - utilising object recognition software to discriminate likely target subjects of interest by their shapes, or - utilising object recognition software to discriminate likely target subjects by colour analyses. According to another aspect of the invention there is provided apparatus for 15 processing and presenting thermographic images comprising means operative to perform each of the steps of the method according to the first aspect. Although the method and system of the present invention will be described herein in both general and in some specific terms, the accompanying drawing will further aid understanding of the method and system. In the drawing: 20 Fig. I shows schematically a camera in use and the processing system and its main functional blocks. The drawing schematically illustrates a digital camera positioned to capture images of a target subject, namely the race horse 12 illustrated approaching the camera. The signals from the camera are supplied to the system 20 which has modules which are shown in 8 functional blocks. The operation of these blocks, which can mostly be implemented by programmed computer, can be understood from. the following description. The digital camera 10 has both infrared and visible spectrum sensitivity, and is located and directed at the subject field 11 which dynamic activities of target subjects 12 5 occur. The camera generates digital image data which comprises data for at least one image frame and which further comprises for each pixel of the or each image frame both infrared and visible spectrum data. The image data for the pixels of the or each image frame is input at 14 to the processing system 20 which has frame capture and frame processing modules 16 and 18 respectively. 10 A thermal input at 21 to the processing system, the thermal input comprising a specification or selection oF isotherm parameters which comprise an infrared or temperature value or a range of infrared or temperature values. The processing system 20 has module 24 operative to identify pixels of the or each image frame having infrared data representing infrared or temperature values of the 15 target subjects satisfying the specified or selected isotherm parameters so as to thereby create a selected pixel set shown at module 26. Module 30 then applies a visibility enhancing false colour value to image data of each pixel in the selected pixel set, and display module 40 can then visibly display the or each image frame so that the pixels of the selected pixel set representing elements of the view of the subject field captured by 20 the camera exhibiting infrared emissions satisfying the isotherm parameters are visibly displayed in an enhanced or false colour superimposed on a visible spectrum image composed of pixels oFtbe image frame other than those in the selected pixel set. The step of providing and directing the digital camera 10 at a subject field 11 may comprise mounting the camera so as to view the subject field either from a fixed or From 25 a selectively variable mounting. For example, the camera may be mounted so as to view 9 a race track, particularly a part of the race track in which activity of most interest to viewers occurs, such as the final stages of a horse race track along which the horses 12 travel towards and eventually reaching the line at the end of the horse race event. For example, a camera may be directed generally longitudinally along the usually straight 5 final section of a horse race track leading up to and including the finishing post. In this way the camera may be directed so as to avoid or at least minimise as much as possible images of subjects or events not of interest. In the particular case of viewing and capturing images of a race horse, if possible it is desirable to exclude images of spectators or onlookers at the race track as they will generate infrared emissions io potentially distracting and possibly even creating spurious infrared signals of no interest or amusement value. In the preferred method, the module or functional block 22 enables the step of providing the thermal input to the processing system 20 comprising a specification or selection of isotherm parameters may be enabled by generating by means of the 15 processing system a visual display of selectable thermal parameters, such as a scale of infrared or temperature values, which an operator can view and utilise to select desired isotherm parameters. For example, the operator can point and click. using computer peripheral devices such as a mouse or touch screen display, whereby the selected temperature or infrared value determines the isotherm parameters for use in the analysis 20 of infrared image data for image frames captured. Alternatively, the operator may select from a display of presented infrared values a range of IR values or temperature values, e.g. by dragging a cursor across or along displayed infrared values, the start and end values of the selected range being defined by first operation of and by subsequent release of the operating switch or button provided by the computer mouse. As a further 25 alternative possibility the specification or selection of isotherm parameters can be by the 10 operator firstly defining a particular range or width of infrared or temperature values and then subsequently selecting a particular IR or temperature value with the defined range or bandwidth of IR or temperature so that the selected values are automatically straddling the selected single value. 5 Whatever method of input of a specification or selection of isotherm parameters is used, the effect on the displayed image at module 40 in which pixels having infrared data representing infrared or temperature values satisfying the isotherm parameters are displayed in enhanced false colours can provide visual feedback to the operator to shift the TR or temperature value up or down the scale, or alternatively or in addition to narrow 10 or broaden the bandwidth of the selected IR or temperature values. The aim is to optimise the information presented, the aesthetic appeal, or other entertainment or amusement value of the displayed image. In summary, therefore, the system can provide an operator with a number of options for input of the isotherm parameters, the options including: 15 - point to and click on a displayed TR/temperature value, touch the screen on which the displayed scale of IR/temperature values is presented to thereby select the desired value, and - point and click on, or touch the screen at, part of the displayed image so that the JR value at that part of the image becomes the central JR/temperature value and a range 20 of IR/temperature values surrounding that value is automatically selected. In this third option, the system may be programmed and operated to present to the operator an image of a target subject within the subject field whereupon the operator can print and click on. a single point or a general area of the displayed image, whereupon the TR values of pixels at that part of the display image can be read or analysed by the system 25 and that particular IR value can then be selected as constituting the selected isotherm I I parameter. For example, only, a displayed image of a racehorse that has entered the subject field and images of which have been captured by the camera can be presented, in real time if desired, to an operator who, for example, points and clicks on a muscular part of the horse's body such as the upper foreleg, whereupon the system adopts the IR data 5 value for that part of the captured image as the specified or selected IR or temperature value that constitutes the isotherm parameters for further processing the image data. It will be seen, therefore, that in real time if desired, a dynamic sequence of presented image frames can be continuously generated by the camera, presented to an operator. and under the real time control of the operator, isotherm parameters can be 10 selected and the subsequent digital image data processed to present images in which infrared emissions from target subjects within the images being captured by the camera which satisfy the selected isotherm parameters are visibly displayed in an enhanced or false colour superimposed on a visible spectrum image of the dynamic activities of the target subjects. In the particular illustrated example, hotter parts of the racehorse's body 15 can be identified and, in real time if desired, allocated enhanced false colour whereby the heat output of the horses participating in the race can be presented to viewers such as television viewers, with informative, educational, or entertainment value or appeal for the viewers. Tn a particular test of the method and system conducted by the applicant, captured 20 images of a horse race which were processed and enhanced according to the present invention showed in one particular event a horse leading upon entering the final straight leading to the finishing post was showing significantly greater IR emissions than other horses at the time of entering the final straight. That fact seemed to indicate that the particular racehorse had expended considerable energy up to that point in the race, 25 greater than the other horses. This conclusion was drawn by the fact that the leading 12 horse showing the higher IR emissions weakened and was overtaken by many of the other racehorses in the final stages of the race. The display of the theri-nographic images of that particular stage of the horse race therefore was useful to present information, education, and entertainment to viewers such as a television audience, particularly if 5 presented accompanied by commentary by expert horseracing commentators or analysts. Although real time image capture, isotherm parameter selection and input, and image processing to produce enhanced or false colour presentation of pixels satisfying the input isotherm parameters has been mentioned as an exercise or process carried out in real time, it needs to be understood that the invention is not so limited. For example, the 10 method may comprise generating the image data for multiple frames and feeding the image data to a storage or memory associated with the processing system. Subsequently, the image data can be presented to an operator who can select different possible isotherm parameters and for each selected set of parameters can process the image data for multiple image frames to see which isotherm parameters produce the most informative or 15 entertaining images. This optimised display can then be presented in the manner of a "replay" of the activities of the target subjects. Whether processed and presented e.g. to a television audience, as a real time display or an optimised replay, the preferred method involves presenting the image frames as a sequence or succession, with each of the multiple image frames having been 20 analysed and presented to the viewers with the visibility enhanced false colour presentations of pixels with IR values satisfying the isotherm parameters, whereby a video or movie of the IR content of interest arising from the target subjects is presented in a visibly enhanced manner for information, education, entertainment or amusement of the viewers. As mentioned above, the displayed sequence of image frames may be a 13television presentation accompanied if desired by relevant commentary on the significance of the IR emissions. Although examples have been given of capturing, analysing and presenting infrared emissions from horses engaged in horseracing events, it will be appreciated that 5 the method and system of the present invention can likewise be utilised in other analogous fields such as analysing, enhancing and presenting IR emissions of horses engaged in harness racing events, IR emissions of greyhounds engaged in races, IR emissions of athletes (although with this possibility, the costumes or clothing may have significant effects upon the detection of I R emissions and hence may diminish the 10 comparative value of viewing, considering, commenting on or learning from IR emissions from one athlete compared to another athlete engaged in the same race or other activity). Other possible fields include capture, processing and presentation of IR data from motor vehicles engaged in racing, or likewise racing motorcycles, with the IR emissions of interest comprising, for example, emissions emanating from vehicle tyres, 15 brake components, vehicle engines, etc. Because the method and system of the present invention may find applications in capturing and processing and presenting of images of dynamic activities where there are many potential sources of systematic or random IR emissions, the method and system of the present invention may further include frame processing to discriminate or enhance 20 the dynamic activities of target subjects as distinct from extraneous, superfluous, background, or transient sources or events producing IR emissions. For example, images of horse races travelling along a race track may include image frame sections from background sources such as spectators located behind or beyond the race track along which the horses are travelling. Other extraneous or transient background sources of IR, 14 apart from people, can include motor vehicles, sunlight reflections from objects, and even momentary bright IR emissions from people operating caineras with flashes. The frame processing to enable removal of, reduction of, or discrimination from unwanted or uninteresting TR sources may comprise for example defining within the or 5 each image frame a limited area within which the selected pixel set is to be confined. That is, by defining for the or each captured image frame a part only of the entire frarme area as being of interest, only IR data satisfying the isotherm parameters within that limited area is enhanced and displayed in the false colour, while TR emissions, even if satisfying the isotherm parameters, from areas outside the defined frame parts will not be 10 enhanced and displayed in the false colour. The definition of the particular frame parts may be under operator control. For example, an operator may view one or a sequence of image frames and may define the frame parts of interest within the or each frame. This may be done, for example, by pointing and clicking at a part of the image frame and the system operates so that only 15 pixels within a predetermined radius of that central pixel would be analysed in accordance with the invention. Alternatively, the operator may click and drag to create a frame or box in each image within which the image data analysis is performed to create the selected pixel set lor IR colour enhancement. In a yet further embodiment, potentially most useful if the digital camera is either 20 Fixed or is movable in a known or repeatable panning movement, images of the subject field can be captured shortly before the dynamic activities of interest occur. These captured background images can be stored and compared to the corresponding captured image frames during the course of the dynamic activities of the target subjects. By processing the captured image frames of interest so as to compare them with stored 25 corresponding image frames, the dynamic activities of the target subjects can be 15 discriminated against background and only pixels representing captured image data for the target subjects can then be analysed to identify the pixels constituting the selected pixel set. Further image data processing refinements can be implemented to further 5 alternatively or to enhance the discrimination of the dynamic activities of the target subjects from background, from extraneous IR sources, from transient IR emissions of no interest, etc. Examples of such further image processing might include limiting the pixel data analysed after preliminary discrimination by one of the previously described methods to contiguous pixels provided there are sufficient numbers of such pixels to 10 validly represent a target subject of interest - this processing can help to eliminate smaller background JR sources Another possible discrimination process might involve comparing corresponding areas of successive captured image frames to ensure that there is a continuity of IR emissions throughout a number of successive captured image frames sufficient to 15 discriminate and therefore eliminate transient IR sources such as momentary sunlight reflections. A yet further frame processing operation to discriminate targets of interest might comprise utilising target subject discrimination such as discriminating movement of the target subjects or using object recognition software to discriminate from shapes or 20 colours likely subjects of interest and eliminate from further processing image frame data from areas of captured image frames other than those discriminated or recognised as target subjects of interest. Detecting or identifying bodies changing relative position within successive captured image frames enables identification of moving target subjects and discriminating them from background or other objects not constituting target subjects 25 of interest. Software applications which generate motion templates, which operate within 16 images or regions of interest within images, which perform pattern recognition (e.g. Haar classifiers), which perform filtering functions, etc., are available commercially for some of the kinds of image processing operations useable in implementing the present invention. 5 It will be appreciated and understood from the preceding description that the method and system of the present invention can enable real time or replay of movie or video images of events and activities in which infrared emissions of subjects can be discriminated, enhanced and displayed, e.g. in a television broadcast, in a manner that provides information, education, entertainment and amusement either alone or in conjunction with 10 learned commentary. Activities such as horseracing events may be particularly enhanced for viewers of broadcasts of such events. It is to be understood that various alterations., modifications and/or additions may be made to the features of the possible and preferred embodiment(s) of the invention as herein described without departing from the spirit and scope of the invention, 15 17