AU2009266403A1

AU2009266403A1 - Method and system for generating text

Info

Publication number: AU2009266403A1
Application number: AU2009266403A
Authority: AU
Inventors: Timothy Lavers; Lindsay Peters
Original assignee: Pacific Knowledge Systems Pty Ltd
Current assignee: Pacific Knowledge Systems Pty Ltd
Priority date: 2008-07-02
Filing date: 2009-02-06
Publication date: 2010-01-07
Also published as: EP2308024A1; EP2308024A4; WO2010000014A1

Description

WO 2010/000014 PCT/AU2009/000134 METHOD AND SYSTEM FOR GENERATING TEXT Field of the Invention 5 The present invention generally relates to a method and system for generating text, and particularly but not exclusively to a method and system for generating syntactically correct text for a test report. 10 Background of the Invention Complicated investigations commonly performed by professionals, such as medical practitioners, often require a large number of tests. The interpretation of is the test results is often difficult and requires the skill of an expert or expert system. The expert or expert system will generate text for inclusion in a report containing a useful analysis and interpretation of the test results, sometimes in a highly condensed form, to be 20 forwarded to the professional. To date, the Knowledge Bases of expert systems have been built in domains in which tests are relatively independent of each other. For example, a thyroid reporting Knowledge Base principally considers TSH, FT3 and FT4 results. Other patient 25 demographic data such as age and sex also generally needs to be taken into account. Reports generated by these Knowledge Bases refer to these individual tests and their values, as well as providing a diagnosis. Typically in these domains, there are less than 20 tests to consider, 30 plus patient demographic data like age and sex, plus clinical notes provided by the medical practitioner. These test typically do not interact. Specific rules are written for each type of 35 investigation. For example, for a thyroid panel of tests, the comment may be generated "Consistent with primary hypothyroidism" if the TSH test result is elevated. Traditional clinical domains have just a few 40 attributes, however, newer clinical domains with potentially hundreds or even thousands of possible investigations, the application of specific rules to each type of investigation becomes infeasible. For example, the medical practitioner may request a number of food 45 allergy tests such as peanut, soya, milk, wheat and egg. If soya and milk return very high positive values 25.3 and 30.1, and the other tests are negative, the pathologist will want the report sent back to the doctor to include a comment like: WO 2010/000014 PCT/AU2009/000134 -2 "Very high results were detected for milk (30.1) and soya (24.3)." The rule giving this comment is: s 10 <= milk <= 50, indicating a very high result, and 10 <= soya <= 50, indicating another very high result, and milk > soya, indicating that the milk value 10 should be before the soya in the report, and peanut = 0, and wheat = 0, and egg = 0 15 In this simple example with just 5 allergens tested, the number of combinations of the above comment is 25 = 32 (when neglecting order of importance). Corresponding to each combination of test results there is a different rule. 20 It is clearly not practical to separately define each of the 32 possible combinations of this comment and corresponding rules even for this simple comment - and real-world examples are far more complex than this. 25 In the case of an allergy Knowledge Base, there are literally hundreds of possible tests that can be performed in an investigation, each measuring the same chemical (IgE), with the value of each test indicating the 30 patient's response to a particular allergen. In cases where there are hundreds of tests in an investigation it is impossible for an expert to define all the possible interactions between the test results and 35 provide the multitude of comment variations that an accurate report would require. In these cases, computerised expert systems containing thousands of rules would be essential, however, computational complexity of generating a report that takes into account potentially 40 hundreds of associated tests is beyond the capability of traditional expert systems. For example, if there were four hundred tests and each test had only a binary output, such as "positive" or "negative", then there would be 2 400 possible combinations of test results, each combination 45 requiring an unique reporting text previously generated and stored on a computer system. This does not even begin to account for possible interactions between the tests which greatly complicates the situation. This is not feasible when there are hundreds or more test results.

WO 2010/000014 PCT/AU2009/000134 -3 The variety of cases and their corresponding reports even with a few attributes can be huge, and even more so when the patient's historical information and clinical notes are also taken into account, and where the test results 5 have continuous values, rather than binary values. Summary of Invention According to a first aspect of the invention there is 10 provided a method of generating information from a plurality of data items, the method comprising the steps of: populating an aggregate data item with at least one of the plurality of data items; and is generating the information using the aggregate data item. In an embodiment, the method comprises the prestep of populating each of a plurality of elements of a predefined 20 structure with a corresponding one of the data values, the structure relating each of the plurality of elements to the aggregate data item. The method may comprise populating the aggregate data item with at least one of the plurality of data items in accordance with the 25 structure. In an embodiment, the method comprises working through the structure to determine one or more characteristics of the aggregate data item. 30 In an embodiment, the structure relates a plurality of aggregate data items, and the method comprises working through the structure to determine one or more characteristics of each of the aggregate data items. 35 In an embodiment, the information comprises textual information. The information may be clinical decision support information. Alternatively, the information comprises a machine instruction. 40 In an embodiment, the step of generating the information comprises the step of generating information using a Knowledge Base or Decision Support System. 45 In an embodiment, the step of populating the aggregate data item comprises a step of receiving the plurality of data items. In an embodiment, the textual information may be WO 2010/000014 PCT/AU2009/000134 -4 syntactically and/or grammatically correct. The textual information may be human readable. In an embodiment, the textual information may form at 5 least part of a report. The report may be associated with one or more test results. In an embodiment, each data item may correspond to one of the test results. The test may comprise any one of 10 an allergy test, a leukemia test, a pathology test, a blood test, and any other type of medical test. A data item may correspond to any other information such as gender, age, demographic information or clinical symptoms. is In an embodiment, the aggregate data item comprises data items which are related. In an embodiment, the step of populating the aggregate data item comprises the step of populating the 20 aggregate data item by applying a rule to at least one of the plurality of data items, or another aggregate data item. The rule may be a domain specific rule. The rule may alternatively be a case specific rule. The rule may be to populate the aggregate data item with one or more of 25 the data items. The rule may be to populate the aggregate data item with one or more of the data items that are above a threshold value. The rule may be to populate the aggregate data item with one or more of the data items that are below a threshold value. 30 In an embodiment, the method comprises: (a) populating one or more aggregate data items with at least one of the plurality of data items; (b) populating one or more further aggregate data 35 items with data items from the one or more aggregate data items by applying one or more rules to the one or more aggregate data items; and (c) generating the information using the one or more 40 further aggregate data items. In an embodiment, each of the plurality of data items are associated with an identifier and a value. Each of the plurality of data items may comprise the identifier 45 and the value. The identifier may be associated with a name or label for the data item. In an embodiment the step of generating information comprises the step of including in the information the WO 2010/000014 PCT/AU2009/000134 -5 names or labels of the data items populating the aggregate data item. The step of generating the information may comprise the step of including in the information the values associated with the data items populating the 5 aggregate data item. The step of generating the information may comprise the step of determining the order of the names or labels in the information. In an embodiment, the step of generating information io comprises determining one or more characteristics of the aggregate data item. The step of determining characteristics may comprise one or more of determining the number of data items comprising the aggregate data item, if the aggregate data item is empty, and if the 15 aggregate data item comprises a specific data item, if an aggregate data item does not contain a specific data item, and if aggregate data items share data items. One of the characteristics may be a value. 20 In an embodiment, the step of generating the information comprises the step of including in the information a determined characteristic of the aggregate data item. 25 In an embodiment, the step of populating the aggregate data item may comprise populating the aggregate data item with one or more other aggregate data items. The aggregate data items and each of the one or more aggregate data items may be associated with an aggregate 30 identifier. The aggregate identifiers may each be associated with aggregate names. The step of generating the information may comprise including the aggregate names. The step of including the aggregate names in the information may include the step of determining the order 35 of the aggregate names in the information. In an embodiment, the step of populating the aggregate data item comprises populating the aggregate data item includes the step of operating on two other 40 aggregate data items. The step of operating may comprise one or more of difference, union and intersection of the two other aggregate data items. In an embodiment, the step of populatng an aggregate 45 data item comprises the step of determining which data items comprising another aggregate data item have values in a particular range or from a particular discrete set.

WO 2010/000014 PCT/AU2009/000134 -6 In an embodiment, the step of generating information comprise applying one or more rules to the aggregate data item. The one or more rules may form at least part of a ripple down rules knowledge system. 5 According to a second aspect of the invention there is provided a system for generating information from a plurality of data items, the system comprising: an aggregate data item populater for populating an 10 aggregate data item with at least one of the plurality of data items; and an information generator for generating the information using the aggregate data item. 15 In an embodiment, the information generator is a textual information generator for generating textual information. Alternatively, the information generator is a machine instruction generator for generating a machine instruction. 20 In an embodiment, the system comprising a Knowledge Base or Decision Support System. In an embodiment, the system comprises a data item 25 receiver for receiving the plurality of data items. The information generator may be arranged to generate syntactically and/or grammatically correct textual information. The information generator may be arranged to 30 generate human readable textual information. The information generator may be arranged to generate coded textual information. The coded textual information may be a machine instruction. 35 In an embodiment, the information generator may be arranged to generate textual information forming at least part of a report. In an embodiment, the aggregate data populator may be 40 arranged to populate the aggregate data item by applying a rule to at least one of the plurality of data items. In an embodiment, the information generator is arranged to include in the information a name or label 45 associated with a data item populating the aggregate data item. The information generator may be arranged to include in the information the value associated with a data item populating the aggregate data item. The information generator may be arranged to determine the WO 2010/000014 PCT/AU2009/000134 -7 order of the names or labels in the text. In an embodiment, the information generator is arranged to determine the characteristics of the aggregate 5 data item. The information generator may be arranged to include one or more of determining the number of data items comprising the aggregate data item, if the aggregate data item is empty, and if the aggregate data item includes a specific data item. 10 In an embodiment, the information generator is arranged to include in the information about the determined characteristics of the aggregate data item. is In an embodiment, the aggregate data populator is arranged to populate the aggregate data item with one or more other aggregate data items. The aggregate data populator may be arranged to include in the text a aggregate data item name associated with an aggregate data 20 item. The aggregate data populator may be arranged to determine the order of the aggregate data item names in the text. In an embodiment, the aggregate data item populator 25 is arranged to operate on two other aggregate data items. In an embodiment, the aggregate data item populator is arranged to determine which data items comprising another aggregate data item have values in a particular 30 range. In an embodiment, the information generator is arranged to apply one or more rules to the aggregate data item. 35 In an embodiment, the information generator is arranged to consider local characteristics where the test was performed. 40 According to a third aspect of the invention, there is provided a method of generating information from a plurality of data items, the method comprising the steps of: evaluating an outcome of one or more rules using one 45 or more aggregate data items each comprising one or more of the data items; and generating the information according to the outcome. In an embodiment, the information comprises textual WO 2010/000014 PCT/AU2009/000134 -8 information. Alternatively, the information comprises a machine instruction. In an embodiment, the step of generating the S information comprises the step of generating information using a Knowledge Base or Decision Support System. In an embodiment, the step of evaluating the outcome of one or more rules comprises using characteristics of an 10 aggregate data item as a basis for one or more of the rules. According to a fourth aspect of the invention, there is provided a system for generating information from a 15 plurality of data items, the system comprising: an evaluator for evaluating an outcome of one or more rules using one or more aggregate data items each comprising one or more of the data items; and an information generator for generating the 20 information according to the outcome. In an embodiment, the information generator is a textual information generator for generating textual information. Alternatively, the information generator is 25 a machine instruction generator for generating a machine instruction. In an embodiment, the system comprising a Knowledge Base or Decision Support System. 30 According to a fifth aspect of the invention, there is provided a method of generating information, the method comprising the steps of: receiving a conceptual representation of information 35 including an interpretive portion, the interpretive portion representing an operation on an aggregate data item comprising a plurality of data items; and generating the information from the interpretive portion. 40 In an embodiment, the information is textual information. Alternatively, the information is a machine instruction. 45 In an embodiment, the step of generating the information includes the step of including in the information one or more names or labels associated with each of the data items. The step of generating the information may include the step of including in the WO 2010/000014 PCT/AU2009/000134 -9 information a collective name for the plurality of the data items. The step of including the one or more names or labels may include the step of integrating the information with a literal portion of the conceptual 5 representation of the text. In an embodiment, the conceptual representation is pseudo text. 10 In an embodiment, the step of generating the information may include the step of generating syntactically and/or grammatically correct textual information. is According to a sixth aspect of the invention, there is provided a system for generating information, the system comprising the steps of: a receiver for receiving a conceptual representation of information including an interpretive portion, the 20 interpretive portion representing an operation on an aggregate data item comprising a plurality of data items; and an information generator for generating the information from the interpretive portion. 25 In an embodiment, the information is textual information, in which case the information generator is a textual information generator. Alternatively, the information is a machine instruction, in which case the 30 information generator is a machine instruction generator. In an embodiment, the generator is arranged to include in the information one or more names or labels associated with each of the data items. The generator may 35 be arranged to include in the information a collective name for the plurality of the data items. The generator may be arranged to integrate the information with a literal portion of the conceptual representation of the text. 40 In an embodiment, the generator may be arranged to generate syntactically and/or grammatically correct textual information. 45 In accordance with a seventh aspect, the present invention provides a computer program comprising instructions for controlling a computer to implement a method in accordance with the first aspect of the invention.

WO 2010/000014 PCT/AU2009/000134 - 10 In accordance with an eighth aspect, the present invention provides a computer readable medium providing a computer program in accordance with the seventh aspect of 5 the invention. In accordance with a ninth aspect, the present invention provides a computer program comprising instructions for controlling a computer to implement a 10 method in accordance with the third aspect of the invention. In accordance with a tenth aspect, the present invention provides a computer readable medium providing a is computer program in accordance with the ninth aspect of the invention. In accordance with a eleventh aspect, the present invention provides a computer program comprising 20 instructions for controlling a computer to implement a method in accordance with the fifth aspect of the invention. In accordance with a twelfth aspect, the present 25 invention provides a computer readable medium providing a computer program in accordance with the eleventh aspect of the invention. The term "server" in this specification is intended 30 to encompass any combination of hardware and software that performs services for connected clients in part of a client-server architecture. The client and a server may be separate software running on a single piece of hardware or a plurality of connected pieces of hardware. 35 Brief description of the Figures In order to achieve a better understanding of the nature of the present invention, embodiments of a method 40 and system for generating textual information will now be described, by way of example only, with reference to the accompanying figures in which: Figure 1 is a block diagram of one embodiment of a system for generating textual information; 45 Figure 2 is a flow diagram of one embodiment of a method of generating textual information; Figure 3 is a block diagram of another embodiment of a system for generating textual information; WO 2010/000014 PCT/AU2009/000134 - 11 Figure 4 is a flow diagram of another method of generating textual information; Figure 5 is a flow diagram of another method of generating textual information; and 5 Figure 6 is a schematic representation of one embodiment of a hierarchial relationship for data items and aggregate data items. Detailed Description of embodiments of the invention 10 In an embodiment of the invention a device receives a plurality of data items, each data item corresponding to the result of one of a plurality of tests. Typically, the plurality of tests are used in an investigation of a 15 patient's conditions, such as does the patient have a particular form of disease or allergy. In this embodiment, the device includes information in the form of a predefined structure relating the types of data items and aggregate data items, allowing the device to populate 20 the predefi-ned aggregate data items with one or more of the received data items by applying various rules which processes the received data. In this embodiment, each data item in one of the aggregate data items are relevant biomarkers for a particular disease or allergy, for 25 example. Further aggregate data items may then be populated by other rules acting on the aggregate data items. The further aggregate data items may, for example, include 30 data items that have a significant value. Further rules are then applied to the further aggregate data items. An example rule may include determining whether the number of significant data items in a further aggregate data item exceed a threshold value. The outcome of the rule may 35 indicate a positive test result, in which case appropriate text reporting a positive, or otherwise, test result is generated. The text may be generated in a flexible case-by-case basis, without requiring a rule for each case, through use of aggregate data items. 40 Figure 1 is a block diagram of one embodiment of a system for generating textual information from a plurality of data items and is generally indicated by the numeral 1. The system 1 may comprise any system able to process 45 information, and in this embodiment may be described as a computer system 1 including a computer program residing on computer readable medium 2 comprising instructions for controlling a central processor 4 of the system, the instructions being to implement a method 500 for WO 2010/000014 PCT/AU2009/000134 - 12 generating textual information from a plurality of data items. A flow diagram of the method 500 is shown in figure 2. 5 In an alternative embodiment, one or more machine instructions rather than textual information are generated, and the components of the system 1 are modified accordingly. It is to be understood that the term "textual information" is to be read more broadly to 10 encompass this alternative embodiment where appropriate hereafter. The computer readable medium 2 includes a non volatile memory 2 in the form of a hard drive disk 2 is connected to the processor 4 by a suitable bus 6 such as SCSI. In some embodiments the non volatile memory includes FLASH memory, a CD, DVD, or a USB Flash memory unit, for example. 20 In some embodiments such as that shown in figure 3, the system 3 is an embedded system. The components in figure 3 similar to those of the system 1 of figure 1 are similarly numbered. The embedded system in this embodiment forms part of an instrument 5 for carrying out 25 a test such as a medical test. It will be appreciated that any suitable architecture, such as terminal/mainframe, could be used and not only those illustrated. 30 In the embodiments shown in figures 1 and 3 the hard drive 2 holds one or more data items 8 representing the results from one or more test from an investigation or case. The system 1 is arranged to process the data items 8. The data items 8 may be associated with a medical test 35 such as an allergy test, a leukemia test, a pathology test, a blood test, or any other type of test. The hard drive 2 may alternatively hold data items relating to any expert domain such as one or more of fraud detection, real estate validation, bone mineral density reporting, medical 40 alerts, genomic, molecular, and allergy reporting, allergy reporting and the systems 1,3 and methods 500 described here may be arranged to process such data 8. In the embodiment of figure 1, the system 1 has a 45 data receiver 10 for receiving the data items 8 subsequently stored on the hard drive 2. In an embodiment were the tests have been done remotely from the system 1, for example at a remote site 12, the system 1 may be arranged for connecting to a network 14 to which the WO 2010/000014 PCT/AU2009/000134 - 13 remote site 12 is also connected. The network 14 may be a wide area network such as the internet, although it will be appreciated that the remote site 12 may be far closer, for example, a room adjacent the system 1 in which case 5 the network 14 may be a local area or wireless network such as WiFI or WLAN. Alternatively, in cases as shown in figure 3 where the system 3 is part of a test instrument 5, the data receiver 10 may act as a interface between the processor 4 and the data source 22 such as a sample 10 testing apparatus of the system 3 that performs the physical, chemical or biological test on a sample or other analysis. The processor 4 is programmed as an aggregate data 15 item populater for populating an aggregate data item 24 with at least one of the plurality of data items 8 stored on the hard drive 2. The aggregate data item 24 is in one embodiment, a type of data structure of a memory 20 for processing by the processer 4. The data items 8 may also 20 be stored in the memory 20. The memory 20 in this embodiment comprises one or more of CPU registers, on-die SRAM caches, external caches, DRAM and/or, paging systems, virtual memory or swap space on the hard drive 2, or any other type of memory. However, embodiments may have 25 additional or less memory types as suitable. The processor 4 is programmed to be a textual information generator for generating the textual information using the aggregate data item 24. The textual 30 information generator 4 is arranged to store the textual information 26 in the memory 20. The textual information 26, in this embodiment, represents human readable text that is syntactically and/or grammatically correct. The output of the system 1,3 is the textual information, 35 preferably in a human readable form such as one or more of text printed to a monitor or screen 28, text printed by a printer 30 onto a paper report 32, and an email or other type of electronic message 34 sent over the network 14 to a physician or surgeon's computer 32, for example. The 40 textual information may be some other decision support outcome derived from the data items 8. In one embodiment, a SMS gateway 34 is instructed by the system 1 to send an electronic message, such as an SMS or email, including the textual information 26 in human readable form to a 45 handheld electronic device 36. The device 36 may be a mobile telephone or Blackberry. In this embodiment, the system 1 is arranged to send instructions to send an SMS to a handheld mobile device 36. This is advantageous when a test result is abnormal and requires immediate follow WO 2010/000014 PCT/AU2009/000134 - 14 up. The processor 4 acting as the aggregate data populator is programmed to populate the aggregate data 5 item 24 by applying one or more rules to at least one of the plurality of data items. The rules may form at least part of a ripple down rules knowledge system, as disclosed in the specification of the applicant's US patent 6,553,361 which is incorporated herein by way of 10 reference. The collection of rules is a Knowledge Base that is built up by an expert as described in the US specification. The rules may be domain specific. For example, the rules may be specific to the domain of allergy testing, or the domain of leukemia testing. In 15 some other instances, however, the rule is a rule specific to the case, that is a rule specific to a set of related test results / data items 8. In this case, the system 1 is a Knowledge Base or Decision Support System. 20 In one case, the data items 8 stored on the hard drive 2 have associated name or label parts and value parts as follows: milk, 25; soya, 30; and 25 peanut, 0. Each of the data items milk, soya and peanut are associated with an identifier (milk, soya and peanut) and a value (25, 30 and 0). In this embodiment, each of the 30 data items comprise the identifier and the value. The identifier is, in this example, a name or label for the data item that can be used for generating the textual information 26. An aggregate data item 24 having a name or label very high food allergens may be populated from 35 the above data items 8 by a rule such as: If milk > 25 then include milk in very high food allergens AND If soya > 25 then include soya in very high food allergens AND If peanut > 25 then include soya in very high food allergens. 40 Alternatively, an aggregate data item 24 having a name or label very high food allergens may be populated from the above data items 8 by a rule such as: very high food allergens is food allergens in range (25, 100) 45 The processor 4 is also programmed as an evaluator for evaluating the outcome of the one or more rules, as exampled above, using one or more aggregate data items, such as 24. The textual information generator 4, in the above example, generated textual information for the 50 report 32, for example, according to the outcome of the WO 2010/000014 PCT/AU2009/000134 - 15 rules. It will be appreciated that the processor 4 may test each data item 8 in turn for inclusion in the aggregate 5 data item 24. In this case, one representation of very high food allergens would be: milk, 25; and 10 soya, 30. The textual information generator 4 may be arranged to include in the textual information 26 the name or label associated with a data item 8 populating the aggregate is data item 26. For example, the processor 4 may be asked to form the textual information: Very high results were found for very high food allergens. 20 The processor 4 would generate textual information representing the text: Very high results were found for soya and milk. The textual information generator 4 has determined 25 that soya has a higher value then milk and thus the best way to present this text is to order the names or labels in the text so that soya is superior. Also, the generator 4 has determined that an and should be placed between soya and milk because there are only two items in this 30 aggregate data item 24. If there was a third item in the aggregate data item 24, such as honey with a value of 26, then the generator 4 would know that one grammatically correct text to generate would be: Very high results were found for soya, honey and 35 milk. The textual information generator is arranged, as required, to include in the textual information 26 the value associated with a data item populating the aggregate 40 data item. For example, the above text may instead be Very high results were found for soya (30), honey (26) and milk (25). The above are examples of one commonly required 45 ordering, but there may be others in different circumstances. In some embodiments, the information generated does not generate textual information, but rather one or more WO 2010/000014 PCT/AU2009/000134 - 16 machine instructions. In this case, the system includes a machine instruction generator. The machine instruction can control workflow. For example, if the test results show that no allergens were detected, then the machine 5 instruction may cause the system to automatically send a report without it being checked by a human evaluator, for example. Alternatively, the machine instruction may cause additional tests to be carried out on held samples before the report is generated. 10 The system 1,3 may include a receiver 36 for receiving a conceptual representation of a text. The receiver 36 in this embodiment includes means such as a keyboard connected to the CPU for entry of the conceptual is representation of the text by an operator 36. Using the above example, the conceptual representation entered by the operator 38 is in the form of pseudo text: Very high results were found for very high food allergens. 20 The pseudotext in this example is compact, informal and represents a high-level description of the text desired by the operator 38, but importantly omits details intended for the system 1,3 to evaluate. It is a natural 25 language description of the computational details. Pseudotext is easier for humans to formulate and read than a more technical description of the desired text which may be achieved using programming or scripting languages. 30 The conceptual representation includes an interpretive portion, which in this case is: very high food allergens The interpretive portion represents an operation on 35 the aggregate data item with the name very high food allergens. The textual information generator 4 generates the textual information 26 from the interpretive portion as described elsewhere in this document. The generator 4 is arranged to include in the textual information 26 one 40 or more names or labels associated with each of the data items 8. The generator 4 may be arranged to include in the textual information 26 a collective name for the plurality of the data items. The generator may be arranged to integrate the textual information 26 with a 45 literal portion of the conceptual representation of the text, which in this case is: Very high results were found for soya, honey and milk.

WO 2010/000014 PCT/AU2009/000134 - 17 In the embodiments shown in figures 1 and 3, the textual information generator 4 is arranged to determine the characteristics of the aggregate data item 24. For example, the textual information generator may be arranged 5 to include one or more of determining the number of data items comprising the aggregate data item, if the aggregate data item is empty, and if the aggregate data item includes a specific data item. These are examples of operations on the aggregate data item. For example, 10 textual information 26 is generated from pseudotext such as: Very high results were found for number of very high food allergens food allergens. which becomes: 15 Very high results were found for 3 food allergens. Thus, the textual information generator 4 is arranged to include in the textual information 26 information about the determined characteristics of the aggregate data item. 20 Number of is a type of operation acting on the aggregate data item very high food allergens. The aggregate data populator 4 may be arranged to populate the aggregate data item with one or more other 25 aggregate data items. The aggregate data item may comprise data items which are related, for example, all foods to which a patient is found to be highly allergic to. The aggregate data populator may be arranged to include in the text a aggregate data item name associated 30 with an aggregate data item. The aggregate data populator may be arranged to determine the order of the aggregate data item names in the text. In an embodiment, the aggregate data item populator 4 35 is arranged to operate on two other aggregate data items. For example, one aggregate data item may be the very high result food allergens, and the other may be food allergens of interest. The populater 4 may then generate a new aggregate data item, for example the very high result food 40 allergens of interest, by taking the intersection of the two aggregate data items. Other possible operators include difference, union and intersection. In another embodiment, the aggregate data item populator 4 is arranged to determine which data items comprising another aggregate 45 data item have values in a particular range. In an embodiment, the step of generating the textual information 26 comprises the step of including in the WO 2010/000014 PCT/AU2009/000134 - 18 textual information 26 information about the determined characteristics of the aggregate data item 24. In some embodiments, the textual information 5 generator 4 is arranged to apply one or more rules to the aggregate data items 24 to control program flow. An example logical test associated with such a rule is: If number of Moderate foods > 1 AND if number of Symptoms > 1 AND number of very high foods + number of io foods = 0 It will be appreciated that aggregate data items can in turn be treated as data items for generating textual information 26 and evaluating rules. Populating the is aggregate data 24 item may include populating the aggregate data item 24 with one or more-other aggregate data items, each of which may have an associated aggregate identifier in the form of a name or label. Populating the aggregate data item may be achieved by operating on two 20 other aggregate data items, or through the application of more general rules such as determining which data items comprising another aggregate data item have values in a particular range. The aggregate name or label can then be used in the textual information 26 just as for the case of 25 using data item names in the textual information 26. Again, the order of the aggregate names in the textual information 26 may be determined by the textual information generator 4. 30 In a lengthy report consisting of several report sections (each with an optional heading), the order in which these report sections is presented is an important factor for the professional or end user. That is, the end user wants to see the most important report sections near 35 the top of the report. However, what makes one report section more important than another depends on the particular case that is being interpreted. It is therefore advantageous to order specified report sections using rules that operate on the data in each case. The 40 placement of some other report sections must be fixed, for example a summary report section that is always at the top of the report. Hence the user may be able to define a mixture of both fixed and variable report section orderings. 45 Allergy reporting is a domain where variable report section ordering may be required. There will be at least five separate report sections - corresponding to the comments on the pollen, food, mite, mould and animal WO 2010/000014 PCT/AU2009/000134 - 19 allergen test results. If the food allergy test results are the most significant for a given patient, then the food report section should come before the other four, and so on. The report section corresponding to the least 5 significant test results should be positioned after the others. Furthermore, there are fixed report sections, namely the summary report section which is at the top of the report, and a recommendations report section which is typically at the bottom of the report. 10 Consequently, the system provides means for the operator 38 to define a "derived data item" for each variable report section, using the rules syntax, which assigns a value corresponding to the desired report is section ordering. In the allergy example above, there would be five derived data items, say "pollen_order", "foodorder", "miteorder", "mouldorder" and "animal-order". Pollen-order would be defined as the highest value of any pollen data item, and similarly for 20 the others. The derived data item "pollenorder" is associated with the pollen report section. For each case, the values of the five derived data items will be calculated, and the corresponding report sections will be ordered according to these values. For example, if the 25 case had data items and values: Grass = 50, birch = 20, (pollen) wheat = 5, soya = 15, (food) mould = 2, mite = 1, 30 cat = 62, dog = 49 (animal) then the report sections would be in the following order: Animal, pollen, food, mould, mite. In some embodiments, the system may provide at least 35 one of the following: 0 A RippleDown rule system as the underlying technology to manage the very large Knowledge Bases required. * Facilities to generate coded information that are 40 machine instructions, such as to control a workflow engine, which for example, controls laboratory workflow such as autovalidation and reflexive testing, using coded outputs from the Knowledge Base. 45 0 Natural language syntax for building rule conditions e Insertion of variables into comments that are evaluated by the specific case that is WO 2010/000014 PCT/AU2009/000134 - 20 interpreted. Variables may be defined using aggregate data items. Example 1 5 A first example application is a leukaemia report knowledge base where diagnosis is performed using hundreds of tests whose values are determined by a micro array of hundreds of protein expression or gene expression markers. An expert may build a diagnosis and report Knowledge Base 10 that identifies the subsets of relevant markers, the diagnosis corresponding to this pattern, and comments for those significant subsets in a textual report to the referring medical practitioner. is The array test results are provided as inputs to the Knowledge Base as a plurality of data items and value pairs. The data items are, in this example, labelled CD1 to CD100 to identify them, indicating 100 elements to the array. Real-world examples may contain several hundreds of 20 markers. In this example, a value for one of the data values of less than 50 means that there is no expression of the antibody corresponding to that marker for that patient 25 sample. A value greater than 50 is possibly significant (depending on the values of other markers). A value higher than 100 for a marker indicates a significant expression. The diagnosis of a particular variety of leukaemia 30 can be deduced from the values of specified sub-sets of the 100 data values. For example, a diagnosis of B-cell Chronic Lymphocytic Leukaemia (B-CLL) can be deduced from the 35 significant expression of at least 2 of CD1, CD2, CD3, CD4 and CD5. This diagnosis is supported by the significant expression of any of CD6, CD7, CD8, CD9 and CD10 although these are not diagnostic of B-CLL in themselves. 40 Alternatively, a diagnosis of Acute Myeloid Leukaemia (AML) can be-deduced from the significant expression of at least 2 of CD11, CD12, CD13, CD14 and CD15. This diagnosis is supported by the significant expression of any of CD16, CD17, CD18, CD19 and CD20 although these are not 45 diagnostic of AML in themselves. Five aggregate data items are populated with received data items as specified by an example structure: WO 2010/000014 PCT/AU2009/000134 - 21 1. "BCLL Diagnostic" populated by the data items CD1, CD2, CD3, CD4, CD5 2. "BCLL Supporting" populated by the data items CD6, CD7, CD8, CD9, CD10 5 3. "AML Diagnostic" populated by the data items CD11, CD12, CD13, CD14, CD15 4. "AML Supporting" populated by the data items CD16, CD17, CD18, CD19, CD20 5. "Leukaemia" populated by the aggregate data items 10 BCLL Diagnostic, AML Diagnostic, BCLL Supporting, AML Supporting A schematic of one embodiment of a structure giving the hierarchial relationship for these data items and is aggregate data items is shown in Figure 6. In some embodiments, once the lower levels of the structure is populated, the value or characteristics at the upper levels are calculated. The structure may be stored in the memory 20 or hard drive 2 (or other data storage unit) of 20 the device 1, for example, and interpreted by the CPU 4. The following ranges may also be defined: e "Undetected" defined as the constant 50 "High" defined as the constant 100 25 To represent the significant data items in each set, further aggregate data items are populated by applying the following rules: 30 1. "Significant BCLL Diagnostic" populated by the rule "BCCL Diagnostic in range [ > High]" 2. "Significant BCLL Supporting" populated by the rule "BCCL Diagnostic in range [ > Undetected]" 3. "Significant AML Diagnostic" populated by the 35 rule "AML Diagnostic in range [ > High]" 4. "Significant AML Supporting" populated by the rule "AML Diagnostic in range [ > Undetected]" A BCLL diagnostic comment is given by the following 40 pseudo text: "Pan-B cell antigen expression with <Significant BCLL Diagnostic>, co-expressed with <Significant BCLL Supporting>, typical of B-Cell Chronic Lymphocytic 45 Leukaemia (B-CLL) ." The variable "<Significant BCLL Diagnostic>" is an instruction to list the names and values of the significant BCLL data items, and similarly for the WO 2010/000014 PCT/AU2009/000134 - 22 variable <Significant BCLL Supporting>. In this embodiment, the listed names and values are ordered in terms of decreasing data item value so that the most significant attributes are listed first. 5 A BCLL diagnostic rule triggers the generation of the BCLL diagnostic comment as follows: "number of Significant BCLL Diagnostic >= 2", and 10 "number of Significant -BCLL Supporting >= 1" That is, the comment is generated if there are 2 or more data items in the set Significant BCLL Diagnostic, 15 and 1 or more data items in the set Significant BCLL Supporting. As a second comment example, the AML diagnostic comment is given by the following pseudo text: 20 "Consistent with AML antigen expression based on positive <Significant AML Diagnostic as names >, co expressed with <Significant AML Supporting as names>. Query possible M2 classification." 25 The data item "<Significant AML Diagnostic as names >" is an instruction to the Knowledge Base to list just the names of the significant BCLL data items, and similarly for the variable <Significant AML Supporting as 30 names>. In this embodiment, the listed names and values are ordered in terms of decreasing data item value so that the most significant data items listed first, even though the 35 values will not be shown for this comment. The AML diagnostic rule triggering the generation of the AML comment may be: 40 "number of Significant AML Diagnostic >= 2", and "number of Significant AML Supporting >= 1" The comment is given if there are 2 or more data 45 items in the set Significant AML Diagnostic. This in turn means that there are 2 or more data items in the set AML Diagnostic which have values greater than 100, and at least 1 data item in the set BCLL Supporting which has a value greater than 50.

WO 2010/000014 PCT/AU2009/000134 - 23 Consider the results of the testing for a sample from patient "A" as follows: CD1 0 CD2 95 CD3 190 CD4 150 CD5 260 CD6 0 CD7 90 CD8 60 CD9 0 CD10 15 CD11 30 CD12 60 CD14 20 CD15 0 CD16 0 CD17 0 CD18 50 CD19 50 CD20 40 CD26 to CD100 0 5 These results are sent to an embodiment of the Knowledge Base which evaluates the aggregate data items and evaluates expressions as follows: 10 Significant BCLL Diagnostic evaluates to "CD5, CD3 and CD4" e Significant BCLL Supporting evaluates to "CD7 and CD8" 15 e Both Significant AML Diagnostic and Significant AML Supporting evaluate to null. The Knowledge Base then makes an interpretation 20 according to the rules defined above. The BCLL rule is applicable in this case as there are 3 elements in the Significant BCLL Diagnostic set, and 2 elements in the Significant BCLL Supporting set. 25 The Knowledge Base evaluates the variables in the BCLL comment "<Significant BCLL Diagnostic>" and "<Significant BCLL Supporting>" then gives the evaluated comment: WO 2010/000014 PCT/AU2009/000134 - 24 "Pan-B cell antigen expression with CD5 (260), CD3 (190) and CD4 (150), co-expressed with CD7 (90) and CD8 (60), typical of B-Cell Chronic Lymphocytic Leukaemia (B 5 CLL)" For a second example, consider the test results for patient "B" as follows: CD1 0 CD2 5 CD3 30 CD4 40 CD5 60 CD6 0 CD7 20 CD8 40 CD9 0 CD10 15 CD11 130 CD12 160 CD14 20 CD15 0 CD16 0 CD17 0 CD18 55 CD19 100 CD20 60 CD26 to CD100 0 10 These results are sent to the Knowledge Base which firstly evaluates the aggregate data items as follows: * Both Significant BCLL Diagnostic and Significant is BCLL Supporting evaluate to null. * Significant AML Diagnostic evaluates to "CD12 and CD11" 20 e Significant AML Supporting evaluates to "CD19, CD20 and CD18" The Knowledge Base then makes an interpretation according to the rules above. The AML rule is applicable 25 in this case as there are 2 elements in the Significant AML Diagnostic set, and 3 elements in the Significant AML Supporting set.

WO 2010/000014 PCT/AU2009/000134 - 25 The Knowledge Base then gives the comment: "Consistent with AML antigen expression based on positive CD12 and CD11, co-expressed with CD19, CD20 and 5 CD18. Query possible M2 classification." Example 2 Another example application is an allergy report knowledge base where there are potentially 500 or more IgE io tests that can be performed. The task of the allergy expert is to advise the referring doctor which subset of the tests performed have significant result values for the patient, which test values may not be significant, which tests need to be followed up, and which further tests, out 15 of the 500 possible tests, should be also be performed. One example solution: * From the total collection of possible data item names, to group those data item names into 20 aggregate data items based on domain-specific rules, e.g. significant pollen attributes, and case-specific rules. e To use any of the characteristics of an aggregate data item as the basis for further rules and / or 25 comments. For example, give a particular comment if the number of elements of an aggregate data item is above a certain number, or if the set includes a particular element * To use one or more aggregate data item as 30 variables in a comment, for example, "The dog, cat and peanut allergies are significant" where the phrase "dog, cat and peanut" is an evaluation of the aggregate data item consisting of allergens that are significant for this case. The 35 generic form of the comment may be The {SignificantAllergens} allergies are significant where the set {SignificantAllergens} is itself defined by rules. * To optionally include the values of the data item 40 in the comment, e.g. The dog (102.3), cat (56.4) and'peanut (43.5) allergies are significant. e To appropriately order the data items in an aggregate data item which appears in a comment, e.g. in terms of decreasing attribute value in 45 the case so that the most significant attributes appear first in the comment. e To automatically format the data items into a naturally constructed sentence that is consistent with the rest of the report. For example if 3 WO 2010/000014 PCT/AU2009/000134 26 attributes are significant the format of the set may be "dog, cat and peanut allergies" whereas if only 2 attributes are significant the format of the set may be "dog and cat allergies" 5 To be able to define a grouping of data items based on a previous grouping of data items, e.g. the new aggregate data item could be the difference, union, or intersection of one set from another, or any set operation. This allows 10 the definition of a hierarchy of sets. For example the difference between the set "appropriate tests" and the set "ordered tests" could identify the set of appropriate tests which have not yet been ordered. 15 e To be able to define a comment that uses either individual data items or an aggregate data item containing those attributes as appropriate. For example, to use the term "food allergy" rather than "peanut, soy, milk, egg and peach allergy" 20 if the list of individual data items would be too long for the comment for that particular case. Similarly, to be able to define a comment that uses a super aggregate data item name rather than subset names as appropriate. E.g. to use the term 25 "inhalant allergy" rather than "pollen, animal, mould, . . .allergies" if the list of individual aggregate data items would be too long for the comment for that particular case. 30 Pseudo text The following table defines some of the features of the pseudo text discussed above, where: s, t, x, y, z refer to data items, either primary or 35 derived data items. The letters X, Y, Z refer specifically to aggregate data items (a type of derived data item). The letters a, b refer to numbers or named constants. The letter p refers to a boolean expressions. 40 Syntax Definition Examples {} {x, y, z ... Aggregate data item food defined as an aggregate data item consisting {peanut, egg, milk} of the specified data items.

WO 2010/000014 PCT/AU2009/000134 - 27 in range {x, y, z ... } in range [a, b] Aggregate data item HighFood an aggregate data item consisting defined as of those specified data items [peanut, egg, milk] in range [20, 50] where the value of that data item is in the specified range including endpoints a and b. food in range [20, 50] wherefood is defined as above. {x, y, z ...} in range (a, b) as above, but the values a and b are not themselves included in the range. Mixed open and closed ranges are also possible, that is {x, y, z ...} in range (a, b] and so on Number of Number of {x, y, z } Number of [peanut, egg, milk] in Number of X range [20, 501 The number of data items in an Number ofHighFood aggregate data item Is empty X is empty HighFood is empty True if there are no data items in an aggregate data item Sum of sum of {x, y, z... } sum of [peanut, egg, milk] in range sum of X [20, 50] The arithmetic sum of the values sum of HighFood of the data items in an aggregate data item Union {s, t... } union {x, y...} Food union animal X union Y This is the set union operation in {s, t... } in {x, y...} HighAllergens infood X inY This is the set intersection operation Not in {s, t... } not in {x, y... } HighAllergens not infood X not in Y This is the set difference operation Includes X includes x HighFood includes peanut True if the aggregate data item X contains the data item x Does not include X does not include x HighFood does not include peanut Negation of "includes" WO 2010/000014 PCT/AU2009/000134 - 28 For X for p Aggregrate data item Defines an aggregrate data item X AllergensForInfants defined as subject to a condition p {peanut, egg, milk} for (age < 2) As groups from {X, Y, Z... } as groups from {S, [allergens] as groupsfrom [food, T ... } pollen, mould, mite,] Represents data items in a will appear in a comment as "food comment using the specified and pollen" for example, rather than aggregate names as constituent data item names peanutt (20.0), egg (15.1) and grass (6.4)" As names {x, y, z} as names Food as names X as names will appear in a comment as "peanut and egg" rather than "peanut (20.0) Represents an aggregate data item and egg (15.1)" in a comment without the corresponding data item values Now that embodiments have been described, it will be appreciated that some embodiments may have some of the following advantages: 5 It is possible to process a large number of text results comprising one investigation. * The report presents.the data items in an appropriate order, with a linguistically natural syntax. 10 e The number of specific report variations are essentially infinite due to the number of possible subsets, and the number of possible orderings within each subset. * The number of specific rule conditions that is determine a particular report are also essentially infinite due to the number of patterns in the attributes in the case, and in their values. e The expert is able to build and maintain the 20 knowledge base with a manageable number of rules. * An expert system is provided that can manage large numbers of attributes and the correspondingly large number of report variations. 25 It will be appreciated that numerous variations and/or modifications may be made to the invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. 30 The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive.

Claims

1. A method of generating information from a plurality of data items, the method comprising the steps of: s populating an aggregate data item with at least one of the plurality of data items; and generating the information using the aggregate data item.

2. A method defined by claim 1 comprising the prestep of io populating each of a plurality of elements of a predefined structure with a corresponding one of the data values, the structure relating each of the plurality of elements to the aggregate data item.

3. A method defined by claim 2 comprising populating the 15 aggregate data item with at least one of the plurality of data items in accordance with the structure.

4. A method defined by claim 1 wherein the information comprises textual information. 20

5. A method defined by claim 1 wherein the information comprises a machine instruction.

6. A method defined by any one of the preceding claims wherein the step of generating the information comprises the step of generating information using a 25 Knowledge Base or Decision Support System.

7. A method defined by any one of the preceding claims wherein the step of populating the aggregate data item comprises a step of receiving the plurality of data items. 30

8. A method defined by claim 4 wherein the textual information is syntactically and/or grammatically correct.

9. A method defined by either claim 4 or 8 wherein the textual information is human readable. 35

10. A method defined by any one of claims 4, 8 and 9 wherein the textual information forms at least part of a report.

11. A method defined by any one of the preceding claims wherein the step of populating the aggregate data WO 2010/000014 PCT/AU2009/000134 - 30 item comprises the step of populating the aggregate data item by applying a rule to at least one of the plurality of data items, or another aggregate data item. 5

12. A method defined by any one of the preceding claims wherein each of the plurality of data items are associated with an identifier and a value.

13. A method defined by claim 12 wherein the identifier may be associated with a name or label for the data 1o item.

14. A method defined by claim 13 wherein the step of generating the information comprises the step of including in the information the names or labels of the data items populating the aggregate data item.

15 15. A method defined by any one of claims 12 to 14 wherein the step of generating the information comprises the step of including in the information the values associated with the data items populating the aggregate data item. 20

16. A method defined by either claim 14 or 15 wherein the step of generating information comprises the step of determining the order of the names or labels in the information.

17. A method.defined by any one of the preceding claims 25 wherein, the step of generating the information comprises determining characteristics of the aggregate data item.

18. A method defined by any one of the preceding claims wherein the step of generating the information 30 comprises the step of including in the information about the determined characteristics of the aggregate data item.

19. A method defined by any one of the preceding claims wherein the step of populating the aggregate data 35 item may comprise populating the aggregate data item with one or more other aggregate data items.

20. A method defined by any one of the preceding claims wherein the step of generating the information comprises applying one or more rules to the aggregate 40 data item. WO 2010/000014 PCT/AU2009/000134 - 31

21. A method defined by claim 20 wherein the one or more rules may form at least part of a ripple down rules knowledge system.

22. A method defined by any one or more preceding claims 5 comprising: (a) populating one or more aggregate data items with at least one of the plurality of data items; (b) populating one or more further aggregate data items with data items from the one or more 1o aggregate data items by applying one or more rules to the one or more aggregate data items; and (c) generating the information using the one or more further aggregate data items. 15

23. A system for generating information from a plurality of data items, the system comprising: an aggregate data item populator for populating an aggregate data item with at least one of the plurality of data items; and 20 an information generator for generating the information using the aggregate data item.

24. A method of generating information from a plurality of data items, the method comprising the steps of: evaluating an outcome of one or more rules using 25 one or more aggregate data items each comprising one or more of the data items; and generating the information according to the outcome.

25. A system for generating information from a plurality 30 of data items, the system comprising: an evaluator for evaluating an outcome of one or more rules using one or more aggregate data items each comprising one or more of the data items; and an information generator for generating the 35 information according to the outcome.

26. A method of generating information, the method comprising the steps of: receiving a conceptual representation of information including an interpretive portion, the 40 interpretive portion representing an operation on an aggregate data item comprising a plurality of data items; and WO 2010/000014 PCT/AU2009/000134 - 32 generating the information from the interpretive portion.

27. A system for generating information, the system comprising: 5 a receiver for receiving a conceptual representation of information including an interpretive portion, the interpretive portion representing an operation on an aggregate data item comprising a plurality of data items; and 10 an information generator for generating the information from the interpretive portion.

28. A computer program comprising instructions for controlling a computer to implement a method in accordance with the method defined by claim 1. is

29. A computer readable medium providing a computer program in accordance with the computer program of claim 28.

30. A computer program comprising instructions for controlling a computer to implement a method in 20 accordance with the method defined by claim 24.

31. A computer readable medium providing a computer program in accordance with the computer program defined by claim 30.

32. A computer program comprising instructions for 25 controlling a computer to implement a method in accordance with the method defined by claim 26.

33. A computer readable medium providing a computer program in accordance with the computer program defined by claim 32. 30

34. A method substantially as herein described with reference to the accompanying figures.

35. A system substantially as herein described with reference to the accompanying figures.