JP2014085854A - Similarity evaluation system, similarity evaluation device, user terminal, similarity evaluation method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To mechanically perform finer similarity evaluation when entropy values of entire files are close to each other.SOLUTION: Each of user terminals 2to 2transmits a comparison source file and a comparison target file to a similarity evaluation device 3. The similarity evaluation device 3 divides each of the comparison source file and the comparison target file into a plurality of sections by a prescribed division method and calculates, per section, prescribed entropy values from the divided comparison source file and the divided comparison target file and obtains a determination result showing whether the comparison source file and the comparison target file are similar or not on the basis of comparison source section feature quantities and comparison target section feature quantities. Each of user terminals 2to 2displays the determination result received from the similarity evaluation device 3.

Description

  The present invention relates to a similarity evaluation technique for evaluating the similarity of files using an entropy value.

In the fields of malware detection and DLP (Data Loss Prevention), an entropy value is obtained as a method of measuring file similarity. The calculation of the entropy value is performed according to Equation (1) when ordering is not taken into consideration. When ordering is taken into consideration, it is performed according to equation (2). Hereinafter, the entropy value considering the order is referred to as M1 entropy. In equations (1) and (2), P _i is the probability that the value is i in the target file.

  By expressing the state of data with an entropy value, it is useful for classifying a large amount of information existing in the storage device and for verifying the approximation between specific data. In particular, there are types of malware and anti-forensic techniques that intentionally change the original data and escape detection, so objective and quantitative representation of the approximation to the original file is meaningful for forensic investigations. Is big.

  The entropy value expresses the state of the data. When a part of the target data is changed, there is a correlation between the amount of change in the entropy value from the original data. On the other hand, in the method using the so-called hash function used for data identification in the forensic field, there is no correlation between the change amount of the data and the hash value, so the objective is as to how much the original file has been changed. It is not possible to make reasonable judgments. This property is also an advantage of the hash function in that it is robust against attacks on the calculated hash value, but the entropy value is more advantageous in terms of verifying the approximation of the file. In addition, since the calculation cost of the entropy value is lower than that of the hash calculation, the similarity evaluation using the entropy value is more suitable for quickly verifying a large amount of data.

  In order to more accurately perform the similarity evaluation using the entropy value, a method of weighting the entropy value with the file size has been proposed (see, for example, Non-Patent Document 1). The weighted entropy value is calculated by equation (3).

Similarity using the weighted entropy value, E ₁ is the weighted entropy value of the comparison source file, E ₂ is the weighted entropy value of the comparison destination file, S ₁ is the file size of the comparison source file, S ₂ Is calculated by the equation (4) with the file size of the comparison destination file.

  Equation (4) is a calculation example and does not take a significant value when applied to a normal file. For this reason, for example, the similarity may be calculated as shown in Equation (5).

Guidance Software, “Utilizing Entropy to Identify Undetected Malware”, [online], [October 15, 2012 search], Internet <URL: http: //www.guidancesoftware.com/DocumentRegistration.aspx? Did = 1000017288>

  Since the entropy value merely represents a random state of the data, even if the entropy value or the weighted entropy value is calculated, it is possible that data having completely different contents may have a close entropy value by chance. Conventionally, supplementary similarity evaluation is performed by visual confirmation of data. However, when there are many files to be compared, it is difficult to visually evaluate the similarity of such files. Therefore, another evaluation means for further evaluating the similarity between files having close entropy values is required.

  The present invention has been made in view of the above points, and provides a similarity evaluation technique capable of performing a more detailed similarity evaluation mechanically when the entropy value of the entire file becomes a close value. The purpose is to do.

  In order to solve the above problems, a similarity evaluation system according to an aspect of the present invention includes a user terminal and a similarity evaluation device, and determines whether or not a comparison source file and a comparison destination file are similar. . The user terminal determines whether or not the input unit that transmits the comparison source file and the comparison destination file to the similarity evaluation device and whether the comparison source file and the comparison destination file received from the similarity evaluation device are similar A display unit for displaying the result. The similarity evaluation device divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and compares the section division unit that generates the comparison source division file and the comparison destination division file with the comparison source division file A predetermined entropy value is calculated for each section from each of the destination division files, and a section feature quantity calculation unit that generates a comparison source section feature quantity and a comparison destination section feature quantity, and a comparison source section feature quantity and a comparison destination section feature quantity And a section similarity evaluation unit for obtaining a determination result based on the determination result.

  A similarity evaluation system according to another aspect of the present invention includes a user terminal and a similarity evaluation device, and determines whether or not the comparison source file and the comparison destination file are similar. The user terminal divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, generates a comparison source division file and a comparison destination division file, a comparison source division file, and a comparison destination Calculating a predetermined entropy value for each section from each divided file and generating a comparison section characteristic value and a comparison target section feature quantity, and a comparison source section feature quantity and a comparison target section feature quantity are similar And a display unit that displays a determination result that is received from the similarity evaluation device and that indicates whether the comparison source file and the comparison destination file are similar to each other. The similarity evaluation device includes a section similarity evaluation unit that obtains a determination result based on the comparison source section feature quantity and the comparison destination section feature quantity.

  According to the similarity evaluation technique of the present invention, when the comparison target file is divided into a plurality of sections and the file similarity is evaluated based on the entropy value for each section, the entropy value of the entire file becomes a close value. Even so, a more detailed similarity evaluation can be performed mechanically. This improves the accuracy of the similarity evaluation between the two files.

It is a figure which illustrates the function structure of the similarity evaluation system which concerns on 1st embodiment. (A) It is a figure which illustrates the function structure of a user terminal. (B) It is a figure which illustrates the function structure of a similarity evaluation apparatus. It is a figure which illustrates the processing flow of a similarity evaluation method. It is a figure which illustrates the calculation result of a whole feature-value. It is a figure which illustrates the evaluation result of whole similarity. It is a figure which illustrates the processing flow of rolling hash. It is a figure which illustrates the calculation result of a section feature-value. It is a figure which illustrates the evaluation result of section similarity. It is a figure which illustrates the function structure of the similarity evaluation system which concerns on 2nd embodiment. (A) It is a figure which illustrates the function structure of a user terminal. (B) It is a figure which illustrates the function structure of a similarity evaluation apparatus. It is a figure which illustrates the processing flow of a similarity evaluation method. It is a figure which illustrates the processing flow of a similarity evaluation method.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function in drawing, and duplication description is abbreviate | omitted.

[First embodiment]
<Configuration>
With reference to FIG. 1, the structural example of the similarity evaluation system 1 of this embodiment is demonstrated. The similarity evaluation system 1 includes N (≧ 1) user terminals 2 ₁ ,..., 2 _N and a similarity evaluation device 3. N user terminals 2 ₁ ,..., 2 _N and similarity evaluation device 3 are connected to network 4. The network 4 only needs to be configured so that the connected devices can communicate with each other. For example, the network 4 can be configured by the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like. Each device does not necessarily need to be able to communicate online via a network. For example, information output from the user terminal 2 _n (1 ≦ n ≦ N) is stored in a portable recording medium such as a magnetic tape or a USB memory, and is input to the similarity evaluation device 3 offline from the portable recording medium. You may comprise as follows.

With reference to FIG. 2A, a configuration example of the user terminal 2 _n included in the similarity evaluation system 1 will be described. The user terminal 2 _n includes a control unit 201, a memory 202, a file storage unit 21, an input unit 22, and a display unit 23. The user terminal 2 _n is a special device configured by reading a special program into a known or dedicated computer having, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The user terminal 2 _n executes each process under the control of the control unit 201. The data input to the user terminal 2 _n and the data obtained in each process are stored in the memory 202, and the data stored in the memory 202 is read as necessary and used for other processes. The file storage unit 21 can be configured by, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device including a semiconductor memory element such as a hard disk, an optical disk, or a flash memory.

  With reference to FIG. 2B, a configuration example of the similarity evaluation device 3 included in the similarity evaluation system 1 will be described. The similarity evaluation device 3 includes a control unit 301, a memory 302, an input unit 31, an overall feature amount calculation unit 32, an overall similarity evaluation unit 33, a section division unit 34, a section feature amount calculation unit 35, and a section similarity evaluation unit 36. , An output unit 37 and a feature amount storage unit 38. The similarity evaluation device 3 is a special device configured by reading a special program into a known or dedicated computer having, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The similarity evaluation device 3 executes each process under the control of the control unit 301. Data input to the similarity evaluation device 3 and data obtained in each process are stored in the memory 302, and the data stored in the memory 302 is read out as necessary and used for other processes. The feature amount storage unit 38 includes, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device including a semiconductor memory element such as a hard disk, an optical disk, or a flash memory, middleware such as a relational database and a key / value store, Etc. can be configured.

<Processing>
With reference to FIG. 3, the operation example of the similarity evaluation system 1 of this embodiment is demonstrated.

The file storage unit 21 included in the user terminal 2 _n (1 ≦ n ≦ N) stores at least two files for evaluating the similarity.

The input unit 22 included in the user terminal 2 _n transmits, to the similarity evaluation device 3, a comparison target file group for evaluating the similarity selected from the files stored in the file storage unit 21 according to the user's operation. . The number of files included in the comparison target file group may be two, or may be three or more. In the following description, a case where the similarity between two files is evaluated will be described, and if there are differences in three or more cases, they will be described as appropriate. The two files to be compared are called a comparison source file and a comparison destination file, respectively.

  The comparison target file group transmitted by the user terminal 2n is input to the similarity evaluation device 3 via the input unit 31 (step S31). The overall feature quantity calculation unit 32 included in the similarity evaluation device 3 calculates a predetermined entropy value from each of the comparison source file and the comparison destination file (step S32). The entropy value calculated from the comparison source file is referred to as a comparison source overall feature amount, and the entropy value calculated from the comparison destination file is referred to as a comparison destination overall feature amount. Various types of entropy values can be applied as the entropy value to be calculated. For example, the entropy value calculated by the above equation (1) may be used. The M1 entropy value calculated by the above equation (2) may be used. It may be a file size weighted entropy value calculated by the above equation (3). For example, N-gram entropy may be calculated. N-gram entropy is an entropy value calculated from the number of occurrences of each bit pattern in a file in units of a determined number of bytes (N bytes). When the comparison target file group includes three or more files, a predetermined entropy value may be calculated for all two file combinations.

  FIG. 4 shows an example of the calculation result of the feature amount of the entire file. The “File Name” column is a physical file name of the comparison target file. The “No.” column is a numerical value that uniquely identifies the comparison target file. The “Entropy” column is an entropy value calculated from the comparison target file by the equation (1). The “M1 Entropy” column is the M1 entropy value calculated from the comparison target file by the equation (2). The “WEntropy” column is a file size weighted entropy value calculated from the comparison target file by the equation (3). The “File size” column is the file size in bytes of the comparison target file. In this example, 15 comparison target files are input. For example, No. 1 ae.bmp indicates that the entropy value is 7.36303, the M1 entropy value is 4.758377, the file size weighted entropy value is 87.8226, and the file size is 151,374 bytes.

  The calculated comparison source overall feature quantity and comparison destination overall feature quantity are stored in the feature quantity storage unit 38. The overall similarity evaluation unit 33 calculates the overall similarity from the comparison source overall feature amount and the comparison destination overall feature amount read from the feature amount storage unit 38 (step S331). The calculation of the overall similarity can be performed by, for example, the above equation (5). When the comparison target file group includes three or more files, the overall similarity may be calculated for the combination of all two files.

  The overall similarity evaluation unit 33 compares the calculated overall similarity with a predetermined threshold value, and obtains a provisional determination result indicating whether the comparison source file and the comparison destination file are similar. In the temporary determination result, a value indicating that the files are similar is set when the overall similarity is equal to or greater than the threshold, and a value indicating that the files are not similar is set when the overall similarity is less than the threshold. (Step S332). When the comparison target file group includes three or more files, a value indicating that a similar file exists is set when there is a combination of files whose overall similarity is equal to or greater than the threshold, and the overall similarity is What is necessary is just to set the value which shows that a similar file does not exist when the combination of the file more than a threshold value does not exist.

  FIG. 5 shows an example of the evaluation result of the similarity of the whole file. The “File Name” column is a physical file name of the comparison target file. The “No.” column is a numerical value that uniquely identifies the comparison source file. Columns “1” to “15” indicate comparison destination files. That is, FIG. 5 is a matrix of the overall similarity of the comparison source file and the comparison destination file. For example, the overall similarity between No. 1 ae.bmp and No. 2 af.bmp is 89. When the threshold value for determining the similarity is set to 99, it is determined that the files are similar in the combination of files whose values are shown in bold in the matrix of FIG. For example, No. 1 ae.bmp is determined to be similar to No. 5 imagesCA6MA9NM.bmp, No. 6 imagesCA83PY61.bmp, and No. 8 imagesCAQI7U2A.bmp.

  If the provisional determination result indicates that the comparison source file and the comparison destination file are not similar, a determination result indicating that the comparison source file and the comparison destination file are not similar is output, and the process ends (step S372). . For example, in FIG. 6, if No.1 ae.bmp and No.2 af.bmp are input as comparison target files, the overall similarity is 89, which is less than the threshold value 99. Two files are determined not to be similar. In this case, the fact that the two files are not similar is output and the process is terminated.

  If the provisional determination result indicates that the comparison source file and the comparison destination file are similar, the subsequent processing is continued. When the comparison target file group includes three or more files and it is determined that the files are similar by a combination of a plurality of files, the subsequent processing is repeatedly executed for all the combinations of similar files.

  The comparison source file and the comparison destination file are input to the section dividing unit 34. The section dividing unit 34 divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method. (Step S34). The file after dividing the comparison source file is called a comparison source division file, and the file after dividing the comparison destination file is called a comparison destination division file. As a method of dividing the section, for example, the section can be divided by a predetermined fixed size. For example, the section may be divided every 10K bytes. Further, for example, the division size may be determined by an equal division proportional to the file size of the comparison target file. If it is equally divided in proportion to the file size, it is possible to avoid the occurrence of a small end size section. For example, the division point may be determined by a rolling hash.

  FIG. 6 shows an operation example of section division by rolling hash. When a file to be divided is input, m bytes are read from the binary string of the file (step S341). This binary sequence of m bytes is called a window. M, which determines the size of the window, can take any value. For example, m = 7 may be set. Next, the hash value of the read window is calculated (step S342). As the hash function, for example, a function for adding the values of all the characters constituting the character string, a function for multiplying all the ASCII codes of the respective characters, and the like can be considered. Subsequently, the lower t bits of the calculated hash value are evaluated (step S343). If the lower t bits match a predetermined arbitrary bit pattern, the position of the window is stored as a division point (step S344). Any bit pattern may be used. For example, the lower t bits may be all 1's or all 0's. For example, t may be determined by the following equation, where L is the file size and N is the total number of sections.

here,

Is the floor function, the largest integer not exceeding x.

  Next, it is confirmed whether the window position is the end point of the file (step S345). If it is not the end point of the file, the window position is moved backward by 1 byte from the top, and the processing from step S341 to step S345 is repeated. The above processing is repeated until the window position reaches the end point of the file.

  If a file is divided into sections using a rolling hash, there is a high possibility that a common part can be extracted as a section except for differences between the two files to be compared.

  The comparison source division file and the comparison destination division file are input to the section feature amount calculation unit 35. The section feature amount calculation unit 35 calculates a predetermined entropy value for each section from each of the comparison source split file and the comparison target split file (step S35). The spectrum of the entropy value for each section of the comparison source divided file is called a comparison source section feature, and the spectrum of the entropy value for each section of the comparison target divided file is called a comparison destination section feature. As the entropy value to be calculated, various types of entropy values can be applied in the same manner as the entire feature amount. For example, the entropy value calculated by the above equation (1) may be used. The M1 entropy value calculated by the above equation (2) may be used. It may be a file size weighted entropy value calculated by the above equation (3). For example, N-gram entropy may be calculated.

  FIG. 7A shows an example of the calculation result of the section feature value of the comparison source file, and FIG. 7B shows an example of the calculation result of the section feature value of the comparison destination file. The “File Name” column is the file name of the comparison target file after the section division. The “Entropy” column is an entropy value calculated from the divided file. The “M1 Entropy” column is an M1 entropy value calculated from the divided file. The “WEntropy” column is a file size weighted entropy value calculated from the divided file. The “File Size” column is the file size in bytes of the divided file. In this example, No. 1 ae.bmp is used as a comparison source file, and No. 5 imagesCA6MA9NM.bmp is used as a comparison destination file, and each file is divided by a fixed size of 100,000 bytes. In FIG. 7A, No.1 ae.bmp is divided into 16 ae.001 to ae.016, and the entropy value, M1 entropy value, and file for each of ae.001 to ae.016 after division. The result of calculating the entropy value with size weight is shown. In FIG. 7B, No. 5 imagesCA6MA9NM.bmp is divided into 16 images of IMAGES ~ 1.001 ~ IMAGES ~ 1.016, and the entropy value, M1 entropy value and file for each of IMAGES ~ 1.001 ~ IMAGES ~ 1.016 after division. The result of calculating the entropy value with size weight is shown.

  The comparison source section feature value and the comparison destination section feature value are input to the section similarity evaluation unit 36. The section similarity evaluation unit 36 obtains a determination result indicating whether the comparison source file and the comparison destination file are similar based on the comparison source section feature quantity and the comparison destination section feature quantity (step S36). As shown in the following equation (6), the determination method calculates the difference between the entropy value for each section included in the comparison source section feature and the entropy value for each section included in the comparison target section feature. Then, the average may be calculated.

Here, n is the total number of sections, H1 _i is the entropy value of section i of the comparison source file, and H2 _i is the entropy value of section i of the comparison destination file.

  If the average D of the differences obtained by Equation (6) is equal to or greater than a predetermined threshold value, it is determined that the comparison source file and the comparison destination file are not in a similar relationship. The threshold for determining whether or not they are similar by the average of the differences may be set to 0.25, for example.

  FIG. 8 shows an example of the evaluation result of the section similarity. FIG. 8A shows the result of calculating the average of the difference between the comparison source section feature value and the comparison destination section feature value. The “section” column is a numerical value indicating the number of the divided section. The “Section entropy value” field is the entropy value for each section of the file to be compared. In the “No.1” column, the entropy value for each section for the comparison source file No.1 ae.bmp is “No.1”. In the column “.5”, the entropy value for each section for No. 5 imagesCA6MA9NM.bmp, which is the comparison destination file, is set. The “difference” column is a value obtained by subtracting the entropy value for each No. 5 section from the entropy value for each No. 1 section. For example, the section entropy value of the divided file ae.001 of No. 1 ae.bmp is 7.190017, the section entropy value of the divided file IMAGES to 1.001 of the imagesCA6MA9NM.bmp of No. 5 is 6.508847, and the difference is 0.68117. It represents that. The “average” row at the bottom of FIG. 8A is the average value of the “difference” column, that is, the average D value of the difference in equation (6). In this example, if the average difference D is 0.472823375 and the threshold value is 0.25 as in the above example, it can be determined that No.1 ae.bmp and No.5 imagesCA6MA9NM.bmp are not in a similar relationship. FIG. 8B shows the result of plotting the comparison source section feature quantity and the comparison destination section feature quantity on a graph. The horizontal axis is the section number, and the vertical axis is the section entropy value. It can be seen that the section entropy values are greatly different in a plurality of sections.

  The determination of whether or not they are similar may be performed using various statistical determinations regarding the comparison source section feature value and the comparison destination section feature value. The statistical determination method may be performed by, for example, standard deviation. That is, the standard deviation of each of the comparison source section feature quantity and the comparison destination section feature quantity is calculated, and if the difference between the standard deviations is less than the threshold value, it is determined that the files are similar. Further, for example, the determination may be made using a correlation coefficient. That is, a correlation coefficient between the comparison source section feature value and the comparison destination section feature value is obtained, and if the correlation coefficient is equal to or greater than a threshold value, it is determined that the files are similar. Further, for example, the determination may be made by comparing the longest match series. That is, the length of the section in which the corresponding section entropy values are continuously matched in the comparison source section feature quantity and the comparison destination section feature quantity is obtained, and the files are similar if the length of the matching section is equal to or greater than a threshold value. judge. For example, the determination may be performed by Fourier analysis. That is, each of the comparison source section feature quantity and the comparison destination section feature quantity is subjected to Fourier transform, and the obtained power spectrum series is compared. If the number of matching elements is equal to or greater than the threshold, it is determined that the files are similar. Further, for example, the determination may be made by the Χ square test. In other words, by calculating the test statistic by approximating the comparison source section feature quantity and the comparison destination section feature quantity to respective predetermined functions, a hypothesis test whether the files are similar with a predetermined threshold as a significance level is performed. Do.

The determination result is transmitted to the user terminal 2 _n via the output unit 37 included in the similarity evaluation device 3 (steps S371 and S371). The information to be transmitted is information necessary for the user terminal 2 _n to display. For example, the total feature amount of the comparison target file, the overall similarity between the comparison target files, the section feature amount of the comparison target file, the similarity determination result between the comparison target files, and the average of the difference of the section entropy values What is necessary is just to select arbitrarily from all the values calculated in order to determine.

The display unit 23 included in the user terminal 2 _n displays the determination result received from the similarity evaluation device 3. The display method may be any method. For example, it may be to format and display the user terminal 2 _n display may be output in a predetermined format to a predetermined printer that is set to the user terminal 2 _n.

<Effect>
In the conventional similarity evaluation technique using the entropy value, the entropy value of the entire file may be close by chance even though the contents of the two files to be compared are completely different. In such cases, conventionally, complementary similarity evaluation has been performed by visual confirmation of data.

  The similarity evaluation system 1 of this embodiment can perform a more detailed similarity evaluation mechanically when the entropy value of the whole file becomes a close value. Therefore, the accuracy of similarity evaluation between two files is improved. Note that the amount of calculation is not greatly increased even by the similarity evaluation technique of the present invention, and a practical level of convenience is maintained.

[Second Embodiment]
<Configuration>
With reference to FIG. 9, the structural example of the similarity evaluation system 5 of this embodiment is demonstrated. The similarity evaluation system 5 includes N (≧ 1) user terminals 6 ₁ ,..., 6 _N and a similarity evaluation device 7. N user terminals 6 ₁ ,..., 6 _N and similarity evaluation device 7 are connected to network 4. The network 4 only needs to be configured so that the connected devices can communicate with each other. For example, the network 4 can be configured by the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or the like. Each device does not necessarily need to be able to communicate online via a network. For example, information output from the user terminal 6 _n (1 ≦ n ≦ N) is stored in a portable recording medium such as a magnetic tape or a USB memory, and is input to the similarity evaluation device 7 off-line from the portable recording medium. You may comprise as follows.

With reference to FIG. 10A, a configuration example of the user terminal 6 _n included in the similarity evaluation system 5 will be described. The user terminal 6 _n includes a control unit 601, a memory 602, a file storage unit 21, an input unit 22, a display unit 23, an overall feature amount calculation unit 32, a section division unit 34, and a section feature amount calculation unit 35. The user terminal 6 _n is a special device configured by reading a special program into a known or dedicated computer having a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The user terminal 6 _n executes each process under the control of the control unit 601. Data input to the user terminal 6 _n and data obtained in each process are stored in the memory 602, and the data stored in the memory 602 is read out as necessary and used for other processes. The file storage unit 21 can be configured by, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device including a semiconductor memory element such as a hard disk, an optical disk, or a flash memory.

  With reference to FIG. 10B, a configuration example of the similarity evaluation device 7 included in the similarity evaluation system 5 will be described. The similarity evaluation device 7 includes a control unit 701, a memory 702, an input unit 31, an overall similarity evaluation unit 33, a section similarity evaluation unit 36, an output unit 37, and a feature amount storage unit 38. The similarity evaluation device 7 is a special device configured by loading a special program into a known or dedicated computer having, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The similarity evaluation device 7 executes each process under the control of the control unit 701. Data input to the similarity evaluation device 7 and data obtained in each process are stored in the memory 702, and the data stored in the memory 702 is read out as necessary and used for other processes. The feature amount storage unit 38 includes, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device including a semiconductor memory element such as a hard disk, an optical disk, or a flash memory, middleware such as a relational database and a key / value store, Etc. can be configured.

Therefore, the difference between the similarity evaluation system 1 of the first embodiment and the similarity evaluation system 5 of the second embodiment is that the similarity evaluation system 1 includes an overall feature amount calculation unit 32 provided in the similarity evaluation device 3. The section dividing unit 34 and the section feature amount calculating unit 35 are configured so that the user terminal 6 _n is provided.

<Processing>
With reference to FIG. 11, the operation example of the user terminal 6 included in the similarity evaluation system 5 of this embodiment will be described.

The file storage unit 21 included in the user terminal 6 _n (1 ≦ n ≦ N) stores a comparison target file group including at least two files whose similarity is evaluated. The number of files included in the comparison target file group may be two, or may be three or more. In the following description, a case where the similarity between two files is evaluated will be described, and if there are differences in three or more cases, they will be described as appropriate. The two files to be compared are called a comparison source file and a comparison destination file, respectively.

The overall feature amount calculation unit 32 included in the user terminal 6 _n calculates a predetermined entropy value from each of the comparison source file and the comparison destination file (step S32). The entropy value calculated from the comparison source file is referred to as a comparison source overall feature amount, and the entropy value calculated from the comparison destination file is referred to as a comparison destination overall feature amount. Since the entropy value to be calculated is the same as in the first embodiment, description thereof is omitted. When the comparison target file group includes three or more files, a predetermined entropy value may be calculated for all two file combinations.

  The comparison source file and the comparison destination file are input to the section dividing unit 34. The section dividing unit 34 divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined method. (Step S34). The file after dividing the comparison source file is called a comparison source division file, and the file after dividing the comparison destination file is called a comparison destination division file. Since the method of segmentation is the same as that of the first embodiment, description thereof is omitted.

  The comparison source division file and the comparison destination division file are input to the section feature amount calculation unit 35. The section feature amount calculation unit 35 calculates a predetermined entropy value for each section from each of the comparison source split file and the comparison target split file (step S35). The spectrum of the entropy value for each section of the comparison source divided file is called a comparison source section feature, and the spectrum of the entropy value for each section of the comparison target divided file is called a comparison destination section feature. The entropy value to be calculated is the same as that of the first embodiment, as is the case with the entire feature amount, and thus description thereof is omitted.

The input unit 22 included in the user terminal 6 _n transmits the comparison source overall feature quantity, the comparison destination overall feature quantity, the comparison source section feature quantity, and the comparison destination section feature quantity to the similarity evaluation device 7.

  With reference to FIG. 12, the operation example of the similarity evaluation apparatus 7 contained in the similarity evaluation system 5 of this embodiment is demonstrated.

The comparison source overall feature quantity, comparison destination overall feature quantity, comparison source section feature quantity, and comparison destination section feature quantity transmitted by the user terminal 6 _n are input to the similarity evaluation device 7 via the input unit 31 ( Step S31). The input comparison source overall feature quantity, comparison destination overall feature quantity, comparison source section feature quantity, and comparison destination section feature quantity are stored in the feature quantity storage unit 38. The overall similarity evaluation unit 33 calculates the overall similarity from the comparison source overall feature amount and the comparison destination overall feature amount read from the feature amount storage unit 38 (step S331). Since the calculation of the overall similarity is the same as that in the first embodiment, the description thereof is omitted. When the comparison target file group includes three or more files, the overall similarity may be calculated for the combination of all two files.

  The overall similarity evaluation unit 33 compares the calculated overall similarity with a predetermined threshold value, and obtains a provisional determination result indicating whether the comparison source file and the comparison destination file are similar. In the temporary determination result, a value indicating that the files are similar is set when the overall similarity is equal to or greater than the threshold, and a value indicating that the files are not similar is set when the overall similarity is less than the threshold. (Step S332). When the comparison target file group includes three or more files, a value indicating that a similar file exists is set when there is a combination of files whose overall similarity is equal to or greater than the threshold, and the overall similarity is What is necessary is just to set the value which shows that a similar file does not exist when the combination of the file more than a threshold value does not exist.

  If the provisional determination result indicates that the comparison source file and the comparison destination file are not similar, a determination result indicating that the comparison source file and the comparison destination file are not similar is output, and the process ends (step S372). .

  If the provisional determination result indicates that the comparison source file and the comparison destination file are similar, the subsequent processing is continued. When it is determined that the files are similar in a combination of a plurality of files, the subsequent processing is repeatedly executed for all the similar file combinations.

  The comparison source section feature value and the comparison destination section feature value are input to the section similarity evaluation unit 36. The section similarity evaluation unit 36 obtains a determination result indicating whether the comparison source file and the comparison destination file are similar based on the comparison source section feature quantity and the comparison destination section feature quantity (step S36). Since the determination method is the same as that of the first embodiment, description thereof is omitted.

The determination result is transmitted to the user terminal 6 _n via the output unit 37 provided in the similarity evaluation device 7 (steps S371 and S371). Since the information to be transmitted is the same as in the first embodiment, description thereof is omitted.

The display unit 23 included in the user terminal 6 _n displays the determination result received from the similarity evaluation device 7. Since the display method is the same as in the first embodiment, the description is omitted.

<Effect>
By configuring as described above, the similarity evaluation system 5 of this embodiment is configured such that data transmitted from the user terminals 6 ₁ ,..., 6 _N to the similarity evaluation apparatus 7 via the network 4 is data. Only the feature quantity with a small data size can be used instead of the evaluation target file with a large size. Thereby, the traffic flowing on the network 4 can be suppressed and the response can be improved.

[Program, recording medium]
The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above-described embodiments are not only executed in time series according to the order described, but may be executed in parallel or individually as required by the processing capability of the apparatus that executes the processes.

  When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

DESCRIPTION OF SYMBOLS 1,5 Similarity evaluation system 2,6 User terminal 3,7 Similarity evaluation apparatus 4 Network 21 File storage part 22 Input part 23 Display part 31 Input part 32 Whole feature-value calculation part 33 Whole similarity evaluation part 34 Section division Unit 35 section feature amount calculation unit 36 section similarity evaluation unit 37 output unit 38 feature amount storage unit 201, 301, 601, 701 control unit 202, 302, 602, 702 memory

Claims (11)

A similarity evaluation system that includes a user terminal and a similarity evaluation device, and determines whether the comparison source file and the comparison destination file are similar,
The user terminal is
An input unit that transmits the comparison source file and the comparison destination file to the similarity evaluation device;
A display unit that displays a determination result received from the similarity evaluation device and indicating whether or not the comparison source file and the comparison destination file are similar;
Including
The similarity evaluation device includes:
A section dividing unit that divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and generates a comparison source split file and a comparison destination split file;
A section feature quantity calculation unit that calculates a predetermined entropy value for each section from each of the comparison source division file and the comparison destination division file, and generates a comparison source section feature quantity and a comparison destination section feature quantity;
A section similarity evaluation unit for obtaining the determination result based on the comparison source section feature quantity and the comparison destination section feature quantity;
A similarity evaluation system characterized by including A similarity evaluation system that includes a user terminal and a similarity evaluation device, and determines whether the comparison source file and the comparison destination file are similar,
The user terminal is
A section dividing unit that divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and generates a comparison source split file and a comparison destination split file;
A section feature quantity calculation unit that calculates a predetermined entropy value for each section from each of the comparison source division file and the comparison destination division file, and generates a comparison source section feature quantity and a comparison destination section feature quantity;
An input unit for transmitting the comparison source section feature quantity and the comparison destination section feature quantity to the similarity evaluation device;
A display unit that displays a determination result received from the similarity evaluation device and indicating whether or not the comparison source file and the comparison destination file are similar;
Including
The similarity evaluation device includes:
A section similarity evaluation unit for obtaining the determination result based on the comparison source section feature quantity and the comparison destination section feature quantity;
A similarity evaluation system characterized by including The similarity evaluation system according to claim 1,
The similarity evaluation device includes:
Calculating the entropy value from the entire files of the comparison source file and the comparison destination file, and generating an overall feature amount calculation unit that generates the comparison source overall feature amount and the comparison destination overall feature amount;
An overall similarity evaluation unit for obtaining a provisional determination result indicating whether or not the comparison source file and the comparison destination file are similar based on the comparison source overall feature quantity and the comparison destination overall feature quantity;
Including
When the provisional determination result indicates that the comparison source file and the comparison destination file are similar, the section dividing unit, the section feature amount calculating unit, and the section similarity evaluation unit are executed. Similarity evaluation system. The similarity evaluation system according to claim 2,
The user terminal is
Calculating the entropy value from the entire files of the comparison source file and the comparison destination file, and generating an overall feature amount calculation unit that generates the comparison source overall feature amount and the comparison destination overall feature amount;
Including
The similarity evaluation device includes:
An overall similarity evaluation unit for obtaining a provisional determination result indicating whether or not the comparison source file and the comparison destination file are similar based on the comparison source overall feature quantity and the comparison destination overall feature quantity;
Including
The similarity evaluation system, wherein the section similarity evaluation unit is executed when the provisional determination result indicates that the comparison source file and the comparison destination file are similar. A section dividing unit that divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and generates a comparison source division file and a comparison destination division file,
A section feature quantity calculation unit that calculates a predetermined entropy value for each section from each of the comparison source division file and the comparison destination division file, and generates a comparison source section feature quantity and a comparison destination section feature quantity;
An interval similarity evaluation unit for obtaining a determination result indicating whether or not the comparison source file and the comparison destination file are similar based on the comparison source interval feature and the comparison destination interval feature;
Similarity evaluation apparatus including An input unit for transmitting the comparison source file and the comparison destination file to the similarity evaluation device;
A display unit that displays a determination result received from the similarity evaluation device and indicating whether or not the comparison source file and the comparison destination file are similar;
User terminal including The comparison source section feature amount obtained by dividing the comparison source file into a plurality of sections by a predetermined division method and calculating a predetermined entropy value for each section, and the comparison destination file is divided into a plurality of sections by the division method. A similarity evaluation section including a section similarity evaluation unit for obtaining a determination result indicating whether or not the comparison source file and the comparison destination file are similar to each other based on the comparison target section feature value for which the entropy value is calculated apparatus. A section dividing unit that divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and generates a comparison source division file and a comparison destination division file,
A section feature quantity calculation unit that calculates a predetermined entropy value for each section from each of the comparison source division file and the comparison destination division file, and generates a comparison source section feature quantity and a comparison destination section feature quantity;
An input unit for transmitting the comparison source section feature quantity and the comparison destination section feature quantity to the similarity evaluation device;
A display unit that displays a determination result received from the similarity evaluation device and indicating whether or not the comparison source file and the comparison destination file are similar;
User terminal including A similarity evaluation method for determining whether a comparison source file and a comparison destination file are similar,
A user terminal sends the comparison source file and the comparison destination file to the similarity evaluation device; and
The similarity evaluation device divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and generates a comparison source division file and a comparison destination division file.
A section feature quantity calculation step in which the similarity evaluation device calculates a predetermined entropy value for each section from each of the comparison source split file and the comparison destination split file, and generates a comparison source section feature quantity and a comparison destination section feature quantity. When,
Section similarity evaluation for obtaining a determination result indicating whether the comparison source file and the comparison destination file are similar based on the comparison source section feature quantity and the comparison destination section feature quantity. Part step,
A display step in which the user terminal displays the determination result received from the similarity evaluation device;
The similarity evaluation method characterized by including this. A similarity evaluation method for determining whether a comparison source file and a comparison destination file are similar,
A user terminal divides each of the comparison source file and the comparison destination file into a plurality of sections by a predetermined division method, and generates a comparison source division file and a comparison destination division file.
A section feature quantity calculating step in which the user terminal calculates a predetermined entropy value for each section from each of the comparison source split file and the comparison destination split file, and generates a comparison source section feature quantity and a comparison destination section feature quantity; ,
The user terminal sends the comparison source section feature quantity and the comparison destination section feature quantity to the similarity evaluation device; and
Section similarity for obtaining a determination result indicating whether or not the comparison source file and the comparison destination file are similar based on the comparison source section feature and the comparison destination section feature An evaluation step;
A display step in which the user terminal displays the determination result received from the similarity evaluation device;
The similarity evaluation method characterized by including this.   A program for causing a computer to function as the similarity evaluation device according to claim 5 or 7 or the user terminal according to claim 6 or 8.

Images