WO2020023009A1

WO2020023009A1 - Data labelling procedures

Info

Publication number: WO2020023009A1
Application number: PCT/US2018/043304
Authority: WO
Inventors: Daniel Cameron ELLAM; Luca BENEDETTO
Original assignee: Hewlett Packard Development Co LP
Current assignee: Hewlett Packard Development Co LP
Priority date: 2018-07-23
Filing date: 2018-07-23
Publication date: 2020-01-30
Anticipated expiration: 2021-01-23

Abstract

Examples of the present disclosure relate to a method of labelling data, whereby the labelled data is used to train a machine learning procedure.

Description

DATA LABELLING PROCEDURES

BACKGROUND

[0001] A data entry of a data set may be tagged with one or more labels. For example, a data entry may be a photograph of an animal with labels such as type=“dog”, colour=“black”, and action=“sleeping”, and the corresponding data set may be a set of photographs of animals with corresponding labels. Informative labels may provide a deeper insight into the data set.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various features of the present disclosure will be apparent from the detailed description which follows, taken in conjunction with the accompanying drawings, which together illustrate, features of certain examples, and wherein:

[0003] Figure 1 is a schematic representation of an example of a process involving a data labelling procedure;

[0004] Figure 2 is a schematic representation of an example of first sets of features of a data set;

[0005] Figure 3 is a graphical representation of an example of a frequency analysis;

[0006] Figure 4 is a block diagram of an example of a procedure for performing a frequency analysis;

[0007] Figure 5 is an example flowchart of a procedure for labelling data;

[0008] Figure 6 is a block diagram of an example of a data labelling procedure;

[0009] Figure 7 is an example of a schematic diagram of a processor and machine- readable storage medium for performing a data labelling procedure; and

[0010] Figure 8 is an example of a schematic diagram of a computer system for performing a data labelling procedure. DETAILED DESCRIPTION

[0011] A data entry of a data set may be tagged with one or more labels. A machine learning procedure may learn from a labelled data set, in order to classify new un- labelled data.

[0012] Certain examples described herein allow a data labelling procedure to be performed. In particular, frequency analysis may be performed on data entries of a data set, so that, for example, the data entries of the data set can be labelled. Having an expert human labelling data may be costly in terms of time, effort and money. A data labelling procedure may enable efficient and accurate labelling of data, which in turn may reduce the cost of labelled data. The labelled data may be used to train a machine learning procedure.

[0013] In certain examples, the machine learning procedure may be used to determine malicious computer processes. Such machine learning procedures may use a large labelled data set, whereby the cost of labelling the data may become unfeasible. In such situations, a smaller labelled data set may have to be used, which may reduce the quality of the machine learning procedure. Performing a data labelling procedure may allow large data sets used for training machine learning procedures to be produced economically.

[0014] A machine learning procedure may learn from inputted data to create a model or classifier for making predictions on new data. The machine learning procedure is presented with example inputs and desired outputs. The example inputs may be referred to as training data or labelled data and the desired outputs may be the labels of the labelled data. The machine learning procedure may then learn from the mapping of the inputs to the outputs to generate a model. The model may in turn be used to predict an output given new data. A machine learning procedure that uses labelled data may be referred to as a supervised machine learning procedure.

[0015] Figure 1 is a schematic representation of a process 100 involving a data labelling procedure 130. A data set 105 is provided which may comprise a set of data entries 101 , 102, 103. A data entry 101 , 102, 103 may comprise at least a first set of features 1 10 and a second set of features 120.

[0016] In an example according to the present disclosure, a data set 105 comprising first sets of features 1 10 may be used to perform a data labelling procedure 130. The data labelling procedure 130 may perform a frequency analysis of the first sets of features 1 10 of the data set 105 and may label each data entry 101 , 102, 103 of the data set 105 to create a labelled data set 140 comprising labelled data entries 141 , 142, 143.

[0017] The machine learning procedure 150 may regard the labels 145, 146, 147 of the data set 140 as desired outputs in order to train and generate a model 170. The machine learning procedure 150 may train on the labelled data set 140 using the second set of features 120 of the data set. In this way, the machine learning procedure 150 may generate a model 170 that can accurately predict the label 141 , 142, 143 of each data entry from the second set of features of the data entry 101 , 102, 103.

[0018] The machine learning procedure 150 is thus trained on the labels 141 , 142, 143 of the labelled data set 140 and the second set of features 120 of the data set 140 to generate the model 170. Training of the machine learning procedure 150 may continue until a certain criterion is satisfied. For example, the machine learning procedure 150 may continue until a pre-defined time limit has been reached. Alternatively, the machine learning procedure 150 may stop when the generated model 170 reaches a pre-determ ined quality criterion.

[0019] In some examples, the machine learning procedure 150 may train on the labelled data set 140 using the first sets of features 1 10 of the data set and the second set of features 120 of the data set. The machine learning procedure 150 may train on the labelled data set 140 using one or more of the features from the first sets of features 1 10 of the data set and/or one or more of the features from the second set of features 120 of the data set. Training the machine learning procedure 150 may generate a model 170.

[0020] The model 170 may then be used to predict the outcome for a new data set 160. The new un-labelled data set 160 set may comprise first sets of features and second sets of features. Using the second sets of features, the model 170 may make predictions 180 for the new data set 160.

[0021] Figure 2 is a schematic representation 200 of a first set of features 1 10 of a data entry 101 . A data entry 101 may comprise at least a first set of features 1 10. In Figure 2, the feature Fi has the value ai from a set of values ai, bi , ci ... etc. The feature F2 has the value 02 from a different set of values a2, b2, 02... etc. In general, the feature Fi from the first set of features can be said to have the values a\, bi, a... etc.

[0022] In some examples, the data entry 101 may contain information for a computer process, and other data entries may contain information for other computer processes. The data entry 101 may comprise a first set of features 110, whereby the first set of features comprises static features of the computer process. Each static feature may have a corresponding value. For example, for a particular computer process, a value for the static feature‘Process name’ may be‘winlogon.exe’.

[0023] In one example, the data labelling procedure 130 of Figure 1 may be performed in two stages. In a first stage, a frequency analysis of first sets of features of a data set is performed, and in a second stage, the individual data entries are labelled based on the results of the frequency analysis stage. An example first stage is described below with reference to Figures 3 and 4, and an example second stage is described below with reference to Figure 5. In another example, the two stages of the data labelling procedure 130 may be performed in parallel.

[0024] Figure 3 is a graphical representation 300 of an example of a frequency analysis of values of a feature. In performing a frequency analysis, a feature 205 from the first set of features 110 is initially selected. The value of the selected feature 205 can then be determined, for example, by reading the value from a data entry. A frequency analysis may be performed on a feature of the other features 215 from the first set of features. For example, if feature Fi 205 with a value of a\ is selected, a frequency analysis may be performed on the other features 215 from the first set of features.

[0025] Figure 3 shows a frequency analysis of the values of feature F2, when Fi takes the value a\. The process for determining these values and their frequency will be described in more detail later with reference to Figure 4. When feature Fi takes the value a\, the Feature F2 may take the unique values a2, b2, 02, d2, e2, f2 310. The frequency 320 of each unique value 310 of feature F2 is determined for the data set. The frequency 320 of each unique value 310 is the number of occurrences of said unique value across all data entries of the data set, when Fi takes the value a\.

[0026] For the unique values 310 of feature F2, a frequency threshold 330 is determined. The threshold defines the minimum number of occurrences for each unique value that is needed before the unique value is determined to be a‘correct’ value for the feature.

[0027] If the frequency 320 for a unique value 310 equals or exceeds the threshold, the unique value 310 may be determined to be‘correct’. If the frequency 320 for a unique value 310 is below the threshold, the unique value 310 may be determined to be‘incorrect’

[0028] In Figure 3, the frequencies for the unique values 02, d2 and e2 all exceed the threshold 320 and hence the unique values 02, d2 and e2 are determined to be correct values for feature F2. The frequencies for the unique values a2, b2 and f2 are below the threshold and hence the unique values a2, b2 and f2 are determined to be incorrect values for feature F2. Therefore, the frequency analysis has determined a set of correct values (02, d2 and e2) for the feature F2.

[0029] Figure 4 is a block diagram of a procedure 400 for performing a frequency analysis. In block 410, a first feature from the first set of features of the data entry is selected. The first set of features of the data entry may be static features of computer processes.

[0030] In block 420, a first value of the first feature is determined. For example, if the first feature that was selected was F1 , a value of the F1 feature may be ai .

[0031] In block 430, another feature from the first set of features of the data entry is selected. For example, another feature may be F2.

[0032] In block 440, unique values for the other feature from the first sets of features of the data set are identified for data entries of the data set that have the first value of the first feature. In other words, for data entries that have the first value for the first feature, all the unique values for the other feature are identified. In the case when the F1 is ai , another feature may be F2. The F2 feature may have unique values a2, b2, C2... etc.

[0033] In block 450, the frequencies for the unique values for the other feature are determined. Determination of the frequency for a unique value may comprise counting the number of occurrences of the unique value. For the other feature F2, the frequency of the unique values a2, b2, 02... etc. are determined.

[0034] In block 460, a frequency threshold for the other feature is assigned. The threshold determines a minimum number of occurrences for a unique value for the other feature. In some examples, the threshold may be determined by a user of the procedure 400 of the present disclosure.

[0035] In other examples, the correct values for the features may be determined by observing the values for the features in a controlled environment, whereby the controlled environment does not contain malicious computer processes. Therefore, the values for the features of the legitimate computer processes are determined to be correct values.

[0036] In block 470, the unique values for the other feature that exceed the threshold are determined to be correct values. If the number of occurrences of a unique value for the other feature exceeds the threshold, the unique value is determined to be a correct value. If the number of occurrences of the unique value does not exceed the threshold, the unique values is determined to be an incorrect value. As described above in relation to Figure 3, the correct values for F2, may be 02, d2, Q2.

[0037] After the unique values for the other feature have been determined to be either correct or incorrect, the procedure 400 may follow a path that returns to block 430. The procedure 400 may repeat blocks 430 to 470 until all other features from the first set of features have been analyzed. For example, block 430 to 470 may be repeated for features F3, F₄... etc.

[0038] The output of the procedure 400 of Figure 4 is a set of correct values for the other features when the first feature has a particular value. The procedure 400 can be repeated for different values of the first feature in the data set to build up a complete set of correct values for the data set.

[0039] As a result, any one data entry of the data set may have a number of features with values that match the correct values determined in the frequency analysis, and a number of feature with values that do not match the correct values determined in the frequency analysis. The number of correct features may be used as a basis for labelling the data entry of the data set.

[0040] In examples where the data entry comprises static features of computer processes, the correct number of features of the data entry may be used to determine if the data entry is a legitimate computer process or malicious computer process. The data entry may then be labeled accordingly. [0041] Figure 5 is a flowchart of an example of a procedure 500 for labelling data. A data entry 101 of a data set 105 comprises at least a first set of features. In block 510 of the procedure, the data entry 101 is evaluated using the results of the frequency analysis of the first sets of features of the data set 105. In particular, the value of the first feature in the data entry 101 is compared to the value of the first feature from the frequency analysis results to determine the set of correct values for the other features. The values of the other features in the data entry 101 are then compared to each of the corresponding correct values for the other features. If there is a match then that feature is considered to have a correct value.

[0042] In block 520, the other features having correct values in the data entry 101 are counted. The number of correct values may be referred to as NCORRECT. Because there is only one value for each feature, the number of correct values equals the number of features with correct values.

[0043] In block 530, the number of correct values NCORRECT is compared to a minimum number of correct values. The minimum number of correct values may be referred to as NMINIMUM.

[0044] If the number of correct values reaches or exceeds the minimum number of correct values, the procedure 500 follows a path to block 540. In other words, the number of correct values NCORRECT is equal or greater than the minimum number of correct values NMINIMUM.

[0045] If the number of correct values does not reach or exceed the minimum number of correct values, the procedure 500 follows a path to block 550. In other words, the number of correct values NCORRECT is less than the minimum number of correct values NMINIMUM.

[0046] In block 540, the data entry is labelled to indicate the classification of the data entry. In some examples, labelling a data entry of the data set may indicate that the data entry is a legitimate computer process. In order to indicate that a data entry for a given computer process has been evaluated to be a legitimate computer process, the data entry may be labelled‘legitimate’,‘benign’ or Ό’ for example.

[0047] In block 550, the data entry is labelled to indicate the classification of the data entry. In some examples, labelling a data entry of the data set may indicate that the data entry is a malicious computer process. In order to indicate that a data entry for a computer process has been evaluated to be a malicious computer process, the data entry may be labelled‘malicious’,‘malign’ or for example.

[0048] After labeling of the data entry 101 , a labelled data entry 141 is created. Labelling at least one data entry 101 of a data set 105 creates a labelled data set 140 comprising labelled data entries 141 , 142, 143. A machine learning procedure may be trained on the labeled data set 140 using a second set of features of the data set.

[0049] Figure 6 is a block diagram of a data labelling procedure 600. In block 610, a frequency analysis of a first set of features of data entries of a data set is performed. The data set may comprise a set of data entries, wherein a data entry comprises a first set of features and a second set of features.

[0050] In some examples, the first set of features may comprise static features of a computer process. The static features of a computer process may comprise at least a process name, a product name, a description, a base priority of the process, a machine identifier, or a binary indicator that the computer process has a graphical user interface (GUI). The static features may comprise features of a computer process that do not change as the computer process is running.

[0051] In some examples, the second set of features may comprise dynamic features of a computer process. The dynamic features may be, for example, the central processing unit (CPU) usage or the memory usage of the computer process. The dynamic features may comprise features of a computer process that change as the computer process is running. The dynamic features may comprise features that reveal how the computer process interacts with the computer as the computer process is running.

[0052] In block 620, a data entry of the data set is labelled based on the frequency analysis of the first set of features. Labelling the data entries creates a labelled data set of data entries.

[0053] In some examples, labelling a data entry of the data set may indicate that the data entry is a legitimate computer process or a malicious computer process. For example, in order to indicate a data entry for a given computer process has been evaluated to be a legitimate computer process, the data entry may be labelled ‘legitimate’, ‘benign’ or O’. Similarly, to indicate that a data entry for a computer process has been evaluated to be a malicious computer process, the data entry may be labelled‘malicious’,‘malign’ or .

[0054] In block 630, a machine learning procedure is trained on the labelled data set. The machine learning procedure may use the second set of features of the data set.

[0055] In some examples, the machine learning procedure may comprise a machine learning procedure for determining malicious computer processes. The malicious computer processes may be masquerading as legitimate processes, such as core computer processes. Alternatively, the malicious computer processes may be core computer processes with malicious code inserted into them.

[0056] Figure 7 is a schematic diagram 700 of a processor 710 and machine- readable storage medium 720 for performing a data labelling procedure. The machine- readable storage medium 720 comprises computer-readable instructions 730 for performing a data labeling procedure which, when executed by at least one processor 710, cause the at least one processor 710 to perform a procedure according to examples described herein. The computer-readable instructions 730 may be retrieved from a machine-readable media, for example any media that can contain, store, or maintain programs and data for use by or in connection with an instruction execution system. In this case, machine-readable media can comprise any one of many physical media such as, for example, electronic, magnetic, optical, electromagnetic, or semiconductor media. More specific examples of suitable machine-readable media include, but are not limited to, a hard disk drive, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, or a portable disc.

[0057] At block 731 the instructions cause the processor 710 to analyze the frequency of first features of a data set. The first features may comprise static features of a computer process. At block 732 the instructions cause the processor 710 to classify the data set based on the frequency analysis of the data set. Classifying the data set may comprise labelling the data set. At block 733 the instructions cause the processor 710 to use second features of the data set to train a machine learning procedure on the classification of the data set. The second features may comprise dynamic features of the computer process. The machine learning procedure may comprise a machine learning procedure for determining malicious computer processes.

[0058] Figure 8 is an example of a schematic diagram of a computer system 800 comprising a processor 810 and a set of instructions 820. The set of instructions 820 are set to cooperate with the processor 810 to perform a data labelling procedure according to examples described herein.

[0059] At block 821 the instructions 820 cause the processor 810 to evaluate the occurrences of features of data entries of a data set. At block 822 the instructions 820 cause the processor 810 to tag the data entries of the data set with labels based on the occurrence of the features of the data entries. At block 823 the instructions 820 cause the processor 810 to train a machine learning procedure on the labels of the data entries using other features of the data entries of the data set. The machine learning procedure may comprise a machine learning procedure for determining malicious computer processes. The features of a data set may comprise static features of the computer process and the other features of a data set comprise dynamic features of the computer process.

[0060] The preceding description has been presented to illustrate and describe examples of the principles described. This description is not intended to be exhaustive or to limit these principles to any precise form disclosed. Features of individual examples may be combined in different configurations, including those not explicitly set out herein. Many modifications and variations are possible in light of the above teaching, for example, training a machine learning procedure for determining or classifying data, whereby the data is not limited to computer processes.

Claims

1. A method comprising:

performing a frequency analysis of first sets of features of data entries of a data set, the data set comprising data entries, wherein a data entry comprises a first set of features of the data entry and a second set of features of the data entry;

labelling the data entries of the data set based on the frequency analysis of the first sets of features of the data entries of the data set to create a labelled data set of data entries; and

training a machine learning procedure on the labelled data set using the second sets of features of the data entries of the data set.

2. A method according to claim 1 , wherein the machine learning procedure comprises a machine learning procedure for determining malicious computer processes.

3. A method according to claim 1 , wherein the first set of features of the data entry comprises static features of a computer process.

4. A method according to claim 1 , wherein the second set of features of the data entry comprises dynamic features of a computer process.

5. A method according to claim 3, wherein the first set of features of a computer process comprises at least a process name, a product name or a description.

6. A method according to claim 1 , wherein labelling a data entry of the data set indicates that the data entry is a legitimate computer process or a malicious computer process.

7. A method according to claim 1 , wherein the frequency analysis comprises: selecting a first feature from the first set of features of the data entry of the data set; determining a first value for the first feature;

selecting another feature from the first set of features of the data entry of the data set;

identifying unique values for the other feature for data entries of the data set that have the first value for the first feature;

determining the frequency of unique values for the other feature;

assigning a threshold of frequency for the other feature; and

determining the unique values of the other feature that exceed the threshold are correct values for the other feature.

8. A method according to claim 1 , wherein labelling a data entry of the data set comprises:

evaluating a data entry of the data set based on the frequency analysis of the first sets of features of the data set to determine how many features of the data entry have correct values;

labelling the data entry to indicate the data entry is legitimate if the number of correct values of features from the first set of features of the data entry reaches a minimum number of correct values; and

labelling the data entry to indicate the data entry is malicious if the number of correct values of features from the first set of features of the data entry does not reach a minimum number of correct values.

9. A non-transitory machine-readable storage medium encoded with instructions executable by a processor, the machine-readable storage medium comprising:

instructions to analyze the frequency of first features of a data set;

instructions to classify the data set based on the frequency analysis of the data set;

instructions to use second features of the data set to train a machine learning procedure on the classification of the data set.

10. A method according to claim 9, wherein the machine learning procedure comprises a machine learning procedure for determining malicious computer processes.

11. A method according to claim 9, wherein the first values comprise static features of a computer process and the second values comprise dynamic features of the computer process.

12. A method according to claim 9, wherein classifying the data set comprises labelling the data set.

13. A computer system comprising:

a processor;

instructions to cooperate with the processor to:

evaluate the occurrences of features of data entries of a data set;

tag the data entries of the data set with labels based on the occurrence of the features of the data entries; and

train a machine learning procedure on the labels of the data entries using other features of the data entries of the data set.

14. A method according to claim 12, wherein the machine learning procedure comprises a machine learning procedure for determining malicious computer processes.

15. A method according to claim 12, wherein the features of a data set comprise static features of a computer process and the other features of a data set comprise dynamic features of the computer process.