JP4420114B2 - Counting method, counting program, counting device - Google Patents

Description

  The present invention relates to a method for counting the appearance frequency of each value in a data series.

  Means for extracting a value that appears at a certain frequency or more from large-scale time-series data that flows continuously is required in various situations. For example, an IP address that detects the occurrence of an error such as packet loss in traffic is used as a data string, and the number of error detections is counted for each IP address to obtain an IP address or route that causes an error of a certain rate or more. There are cases.

In such a case, the simplest counting method may be to have a counter for each IP address and increase the counter each time an error occurs. However, since the IP address space is enormous and requires counters for at least the connected terminals, a large memory capacity is required.
Also, it is necessary to scan a large memory space in order to search for counters with a certain frequency or more.

In this regard, in Non-Patent Document 1 below, data that appears at a certain frequency or more with a small amount of memory by deleting data with low frequency at any time so as to be stochastically or within a certain error range. And the method of counting the frequency is discussed.
The technique described in Section 4.2 “Lossy Counting Algorithm” of Non-Patent Document 1 below will be briefly described as follows.

(1) N areas for counting the data series are secured and divided into sections each having an inverse capacity of the allowable error rate ε. Numbers starting with 0 are assigned to the sections.

(2) N pieces of data are set in the area, and counting is started in order from the first data (= first data in the first section). Each time a new value appears, the section number Δ at that time is recorded together with the value of the data.
When an already recorded value appears again, the count number f of that value is increased by one.

(3) Every time the boundary of the section is reached, data with a small count number f is deleted from the count result total according to the following criteria.
(3.1) Data with f + Δ ≦ current section number is deleted from the tabulation result.
(3.2) Data that does not satisfy the above equation is left as it is.

When the above steps are repeated until the last data (Nth data), it is mathematically guaranteed that all data exceeding a certain number of counts is retained, and data with a small number of counts is represented by step (3. Deleted in 1) does not remain.
Therefore, only a memory capacity necessary for counting important data having a large number of counts is required, and the appearance frequency of each value in a large amount of data series can be counted with a smaller memory capacity.

Gurmeet Singh Manku, Rajeev Mottani, “Approximate Frequency Counts over Data Stream”, VLDB 2002 (28th VLDB), p. 346-357, August 2002

As in traffic observation, when data continues to flow without termination and the IP address that is actually communicating changes frequently, the IP address that appears frequently in the observed data series changes over time. Often to do.
In this case, it is desirable in terms of traffic observation that data with high frequency can be continuously extracted without being interrupted from the viewpoint of grasping the time-dependent change of the IP address in real time as much as possible.

However, in the technique described in Non-Patent Document 1, since the processing is divided into one when N pieces of data are processed, the next N pieces are processed even when it is desired to continuously extract frequently used data. I have to wait until
For example, the process of extracting high-frequency data from 100,000 data obtained in time series cannot be performed every time 100 new data is obtained. You have to wait until the count is complete.

If an attempt is made to realize the above without waiting for the count of N data to be completed, it is necessary to perform a plurality of processes in parallel by shifting the desired number of data.
In this case, a separate memory area and data processing capability are required for each parallel processing, and a great deal of resources are consumed.

  Therefore, there has been a demand for a counting method capable of sequentially obtaining the progress of counting without executing individual parallel processing while suppressing memory capacity by deleting data having a low statistical appearance frequency.

  A counting method according to the present invention is a method of grouping equivalents of input data series and counting the appearance frequency of the equivalent data for each group, and storing a predetermined number of input data series A storage device having an area is provided, the storage area is divided into reciprocal sections of an allowable count error, and equivalent data is grouped while inputting the data series into the storage area. A counting step for counting the appearance frequency of the equivalent data, and all or part of the counting results counted in the initial stage when the number of the data series input to the storage area reaches the maximum size of the storage area A discarding step for discarding each of the groups, a low-frequency data deleting step for deleting one with a low appearance frequency among the groups, and a group of the data series. An update step for updating the counting result by additionally inputting the number corresponding to the number, and in the counting step, the counting result of each group and the allowable counting error of the group are combined and stored in the storage device. In the discarding step, the count error of the group that discards the count result is updated using the count result of the group before the discard, so that the count error before and after the discard is within the range of the allowed count error. It is to be stored.

According to the counting method of the present invention, data having a low statistical appearance frequency is deleted from the counting result, so that the memory capacity for storing the counting result can be suppressed.
In addition, in the update step, data for one section is additionally input to update the count result, so that the data count result can be obtained sequentially.
In addition, from the viewpoint that the old counting result is not important, when the number of input data reaches the maximum size, all or part of the counting result counted in the initial stage is deleted in the discarding step. There is a saving effect.
Also, when deleting the initial counting result in the discarding step, the allowable counting error is updated based on the counting result before discarding, so that even if the old counting result is deleted, the counting result within a certain error range is maintained. And the accuracy of counting is maintained.

Embodiment 1 FIG.
FIG. 1 is a functional block diagram of a counting device 100 according to Embodiment 1 of the present invention.
The counting device 100 is a device that inputs a data series and counts the appearance frequency of the same data value that appears in the series, and includes a counting unit 110, a storage unit 120, a discard processing unit 130, a low-frequency data deleting unit 140, An update unit 150 is provided.

The counting unit 110 receives the data series, counts the appearance frequency of the same data value, and stores the counting result in the storage unit 120.
The storage unit 120 is a storage device that stores the count result of the data series.
The discard processing unit 130 deletes the counting results counted in the initial stage from the counting results stored in the storage unit 120.
The low frequency data deletion unit 140 deletes the count results stored in the storage unit 120 with a low appearance frequency.
The updating unit 150 sequentially receives additional data series after the counting unit 110 finishes counting a certain number of data series, and stores the count result in the storage unit 120. The counting result previously stored in the storage unit 120 is updated with the additional counting result.
Specific processing procedures of these units will be described with reference to FIGS.

The counting unit 110, the discard processing unit 130, the infrequent data deleting unit 140, and the updating unit 150 can be configured by hardware such as a circuit device that realizes these functions, a CPU (Central Processing Unit), a microcomputer, or the like. It is also possible to configure the computer with software that defines its operation.
All or a part of these components can be configured integrally.
The storage unit 120 is preferably configured from a relatively large capacity storage device such as an HDD (Hard Disk Drive) from the viewpoint of capacity.

FIG. 2 explains the division of the data series counted by the counting unit 110.
When counting the data series, the counting unit 110 sets an input buffer of a certain size on the storage unit 120, and executes the count after taking the data series into the buffer. Here, the following description will be made assuming that a buffer size sufficient to count N data is secured.
Further, for the convenience of the processing procedure described later, the above-described input buffer is divided into a plurality of sections. The division criteria are as follows.

(1) Let ε be an error (allowable error) allowed in counting the data series.
(2) The number of data in one section is (1 / ε).
(3) The total number of sections is εN. A section number starting from 0 is assigned to each section.

The counting unit 110 counts the first N pieces of data, that is, data up to “section number = εN−1”. The update unit 150 counts the data in the subsequent sections.
The counting process performed by the counting unit 110 is performed by inputting N pieces of data to the buffer, but the subsequent counting process performed by the updating unit 150 performs counting for each section of data. Therefore, subsequent count results are obtained sequentially for each section.

  Here, for comparison with the counting device 100 according to the first embodiment, the conventional counting method described in Non-Patent Document 1 will be described. Thereafter, the description returns to the operation of the counting device 100 according to the first embodiment.

  FIG. 8 shows an example of storing count results in the prior art. Here, an example of storage in a table format is shown. It is assumed that the data series is divided into a plurality of sections and counted in the same manner as described with reference to FIG.

The table in FIG. 8 includes a “data value” column, an “appearance frequency” column, and a “counting start position” column.
In the “data value” column, values of data appearing in the data series are stored.
The “appearance frequency” column stores the appearance frequency of the data indicated by the “data value” column.
The “counting start position” column stores the section number when the data shown in the “data value” column first appears in the data series. The value in this column is also a value representing an allowable error in counting the data value. This is because the number of data in one section is (1 / ε).

In the example of FIG. 8, it can be seen that the data of “data value = D1” first appears with “section number = 0”, and 410 data of the same value have been counted so far.
Similarly, it can be seen that the data “data value = D2” appears first with “section number = 10”, and 320 data of the same value have been counted so far.

  In the prior art described in Non-Patent Document 1, data whose counting result satisfies the following (Equation 1) is to be deleted as low-frequency data with low appearance frequency. As a result, only data with a high appearance frequency that seems to be high in importance is left, and the memory capacity is reduced.

  Appearance frequency + statistical start position ≤ current section number (Equation 1)

However, in the prior art described in Non-Patent Document 1, since the input buffer is set to correspond to N data, another application that uses the count result until the N counts are completed. Etc. will be kept waiting.
This is inconvenient for applications that want to obtain counting results in real time or close to this frequency. Therefore, the counting device 100 according to the first embodiment proposes a method capable of sequentially updating the counting result after the first N data has been counted.

However, if the count result is updated sequentially, the parameter of the count fluctuates, so that the above (Formula 1) provided on the assumption that the number of parameters is N cannot be used as it is.
Therefore, in the counting device 100 according to the first embodiment, consideration is given so that the allowable error is updated each time and the above (Equation 1) can be used. Specifically, this will be described with reference to FIGS.

In the above, the counting method in the prior art described in Non-Patent Document 1 has been described.
Hereinafter, the description returns to the operation of the counting device 100 according to the first embodiment.

FIG. 3 illustrates a counting process executed by the counting unit 110.
When counting the appearance frequency of the data in the input data series, the counting unit 110 records the time at which the appearance frequency exceeded a predetermined threshold in the recording unit 120 together with the appearance frequency. In FIG. 3, “εN = 50” is set as an example of the above-described threshold value.
The threshold value set here becomes a reference for a new allowable error. Specific examples will be described later with reference to FIGS.
When counting the appearance frequency of the data “data value = D1”, the counting unit 110 records the section number at that time in the storage unit 120 every time the appearance frequency exceeds the threshold value 50 described above. The storage format will be described later with reference to FIG.

  In the example of FIG. 3, the appearance frequency exceeds 50 at “section number = 5”, and the appearance frequency exceeds 50 again at “section number = 17”, which is 12 sections ahead. The counting unit 110 records these section numbers in the storage unit 120 so that they can be specified later.

FIG. 4 shows an example of storing the count result of the counting unit 110 in the storage unit 120.
4A shows a count result table, and FIG. 4B shows a configuration and data example of a threshold position table.

The count result table is a table for storing the count results of the data series, and has a new “allowable error” column in addition to the table example in the prior art described with reference to FIG.
In the prior art, the allowable error is constant regardless of the progress of the count, but in the first embodiment, the allowable error sequentially varies according to the procedure of FIG. 5 described later, so this column is newly provided.

  In the example of FIG. 4A, the value in the “allowable error” column is the same as the value in the “counting start position” column. This is because the tolerance does not change until the procedure of FIG. 5 described later is executed, and is equal to the initial state (= statistic start position value).

  The threshold position table is a table for sequentially storing the positions where the counting result exceeds the threshold (50), as described with reference to FIG. 3, and includes a “data value” column, a “distance distance until next update” column, and an “allowable” It has an “error update value” column, an “initial interval frequency value” column, and a “data value serial number” column.

In the “data value” column, values of data appearing in the data series are stored.
In the “section distance until next update” column, the number of sections required from when the counting result exceeds the previous threshold value (50) until the next threshold value (50) is stored is stored.
The “allowable error update value” column stores the count results from the count result exceeding the threshold (50) to the next interval immediately before the threshold (50) is exceeded.
If the coefficient result further exceeds the threshold within the same section after storing the value in each column because the threshold is exceeded in the first section (section number 0) or the threshold position table, “until the next update” The “distance interval” column stores “0”, and the “allowable error update value” column stores count results up to the current interval.
The “initial interval frequency value” column stores the appearance frequency of data only in the interval where the counting result exceeds the threshold value (50).
The “data value serial number” column is a number that is newly assigned every time the counting result exceeds the threshold (50), and is provided for convenience in order to identify the recording order of the values in the same “data value” column. Is.

In the example of FIG. 4B, the data in the first and second rows is an example of data that is in line with the example described in FIG.
That is, since the counting result has exceeded 50 for the first time after starting counting from the first interval, “distance distance until next update = 6” is set in the first row, and the counting result up to the immediately preceding interval is Since it is 43, “allowable error update value = 43” is set, and since there is no data regarding “D1” in the threshold position table, “initial frequency value = 0” is set.
Next, the count result exceeded 50 when it was further 12 sections forward, so in the second row, “section distance until next update = 12” and the count result up to the immediately preceding section is 32. “Allowable error update value = 32”, and since the frequency of appearance of data only in the section (section number = 5) exceeding the previous 50 is 8, “initial frequency value = 8”.
Since the first row and the second row are threshold positions for the same “data value = D1”, the values in the “data value serial number” column are incremented by 1 in order from 1, and are assigned numbers.

The “allowable counting error” in the first embodiment corresponds to εN.
The “allowable count error update value” corresponds to the value in the “allowable error update value” column of the threshold position table.

  FIG. 5 illustrates a processing procedure of the discard processing unit 130. Hereinafter, description will be made in the order of steps in FIG. In FIG. 5, colored double-ended arrows indicate the count target range in each step.

(1) Count N The counting unit 110 counts the first N pieces of data and stores the counting result in the storage unit 120. Here, it is assumed that the same counting results as those described with reference to FIGS.

(2) Delete initial counting result The discard processing unit 130 executes the following processing.
(2.1) The discard processing unit 130 deletes all the count results of the data D1 in the sections 0 to 4.
(2.2) Since the same number of errors occur in the counting result of the data D1 due to the deletion of the counting result, the allowable error value related to the data D1 is updated by the same number using the threshold position table described with reference to FIG. . The update procedure will be described later with reference to FIG.

(3) Shifting the section by one The count target range is advanced by one section, and the update unit 150 is responsible for counting the subsequent data.
Here, the data of the section “section number = εN” is counted. Similarly, the update unit 150 thereafter sequentially stores the count results in the storage unit 120 while moving forward one section at a time. Other applications can obtain count results stored sequentially for each section.

(4) Delete the initial counting result When the left end of the counting target section reaches a section (section number = 5) in which the counting result of the data D1 exceeds 50, the discard processing unit 130 executes the following processing.
(4.1) The discard processing unit 130 deletes all the counting results of the data D1 in the sections 5 to 16.
(4.2) Since the same number of errors occur in the counting result of the data D1 due to the deletion of the counting result, the allowable error value related to the data D1 is updated by the same number using the threshold position table described with reference to FIG. . The update procedure will be described later with reference to FIG.

  Similarly, the discard processing unit 130 discards the old count result every time the left end of the count target section reaches the section where the count result of the data D1 exceeds 50, and the same number of allowable error values for the data D1. Update.

FIG. 6 explains the procedure for using the data stored in the threshold position table. The same data values as those described in FIG. 4 were used.
FIG. 6A shows a state before the discard processing unit 130 performs the discard process of FIG.
The discard processing unit 130 checks the value in the “counting start position” column of the counting result table for each section boundary, and searches for data in which the value in the column is “0”. Here, since the data D1 appears from the section number 0, the value in the same column of the data D1 is “0”.

  Next, the discard processing unit 130 searches the data in the threshold position table for “data value = D1”, and acquires the data having the smallest “data value serial number”. Here, the data in the first row in FIGS. 6 (1) and 6 (b) correspond. If such data is not found, it indicates that the count result of “data value = D1” is less than the allowable error “εN = 50”, and therefore the count result of “D1” is deleted from the count result table.

  Next, the discard processing unit 130 updates the data related to the data D1 in the count result table using the data in the first row in FIGS. 6 (1) and 6 (b). Specifically, the following processing is performed.

(1) The discard processing unit 130 subtracts the sum of the value of the “allowable error update value” column and the “initial interval frequency value” column of the threshold position table from the value of the “appearance frequency” column of the count result table. . This is to delete the counting result before the section number 5, and corresponds to the processing of step (2) in FIG.

(2) The discard processing unit 130 updates the value in the “allowable error” column of the count result table with the value in the “allowable error update value” column of the threshold position table. However, when the value of the “distance distance until next update” column is 0, the value of the “allowable error” column is updated with 0. This is a compensation process that accompanies the deletion of the count result of section number 5 or earlier, and is based on the fact that an error corresponding to the deletion occurs in the count result.

(3) The discard processing unit 130 updates the value of the “counting start position” column of the counting result table with a value obtained by subtracting 1 from the value of the “section distance until next update” column of the threshold position table. However, if the value in the “distance distance until next update” column is 0, the value in the “coefficient start position” column is left as 0 (that is, not updated). This is a process for preparing for step (4) in FIG.
Thereafter, every time the counting target section advances by 1, the updating unit 150 subtracts 1 from all the “counting start position” columns in the counting result table, and the data for which the value in the column becomes 0 is Then, the discarding process similar to the above (1) to (2) is executed.
For example, the next execution of the discarding process for the data D1 is the time when the value of the “counting start position” column becomes 0, so it is five sections ahead, that is, the time of step (4) in FIG.

FIG. 6B shows the state of each table after the discard processing unit 130 performs the above processing.
The first row data of the counting result table is updated to delete the old counting result, the value in the “allowable error” column is updated, and the first row data of the threshold position table is further deleted. I understand.

  Thereafter, the discard processing unit 130 repeats the same procedure every time the section boundary is reached. As a result, the data of the counting result table deletes the old counting result and holds only new data, and the value in the “allowable error” column is updated with a value corresponding to the discarded data. The memory capacity is saved and the accuracy of the counting result is also maintained at a certain level.

It should be noted that the old count results are deleted when the data series in which the latest data reaches the counting device 100 one after another is counted, the more old data is, the more the current state is statistical This is in view of the fact that it is unnecessary to grasp the situation.
Referring to FIG. 6, data whose value in the “counting start position” column is 0 is data that appears at the very initial stage of counting. This is because it is considered that there is not much influence on statistically knowing the state of the same data.

In addition to the discard processing unit 130 performing the discard processing described with reference to FIGS. 5 to 6, the low-frequency data deletion unit 140 performs the above (formula 1) as in the conventional technique described in Non-Patent Document 1. Is used to delete data with a low appearance frequency from the count result table.
These double deletion processes can effectively reduce the memory capacity to be consumed, and the update unit 150 sequentially updates data, so that count results can be obtained sequentially at the time interval required for data update for one section. Can do.

  In summary, the operation procedure of the counting device 100 according to the first embodiment is roughly as follows.

(1) The counting unit 110 performs counting after setting the first N pieces of data in the buffer, stores the counting results in the counting result table, and sequentially sets values in the threshold position table.

(2) After the counting unit 110 counts N data, the discard processing unit 130 indicates whether or not to discard old data for each section boundary, and the value of the “statistic start position” column is 0. Is determined, and the discarding process as described with reference to FIG. 5 is performed.

(3) The low frequency data deletion unit 140 deletes the low frequency data that satisfies the condition of (Equation 1) for each section.
(4) After the counting unit 110 counts N pieces of data, the updating unit 150 stores the count result of the additional data for each section in the counting result table and the threshold position table.

  In the first embodiment, the table format as shown in FIG. 4 has been described as an example of the storage format for counting results, but the present invention is not limited to this, and an arbitrary storage format such as a list format may be used. it can.

As described above, in the counting device 100 according to the first embodiment, the counting unit 110 records the section number at the time when the appearance frequency exceeds the threshold (εN = 50) in the threshold position table, and performs the discard process. After the N counts are completed, the unit 130 refers to the threshold position table for each section boundary and deletes the initial count result under a predetermined condition.
Thereby, the memory capacity for storing the counting result can be saved.
Further, since the count result at the initial stage is not so important from the viewpoint of statistically knowing the latest state, the accuracy of the count result is maintained at a certain level even if such deletion processing is performed.

  When the discard processing unit 130 deletes the counting result at the initial stage, the “allowable error update value” in the threshold position table, that is, the counting result until immediately before the counting result exceeds the set threshold (allowable error) is used. Since the value in the “allowable error” column of the counting result table is updated, the counting result within a certain error range is maintained even when the old counting result is deleted, and the accuracy of the counting result is maintained.

In addition, after the counting unit 110 counts N pieces of data, the update unit 150 sequentially stores the count result in the count result table every time the data for one section is counted. The counting result can be obtained in real time or at a time interval close to this.
As a result, an application that wants to sequentially use the counting result does not have to wait for the processing until the counting is completed, thereby contributing to a quick processing.

  Moreover, since the low frequency data deletion unit 140 deletes data with a low statistical appearance frequency from the count result, the memory capacity for storing the count result can be reduced.

Embodiment 2. FIG.
FIG. 7 is a functional block diagram of the packet collection device 200 according to Embodiment 2 of the present invention. The packet collection device 200 is a device for collecting communication packets flowing on the network, counting predetermined information in the packets, for example, transmission / reception addresses, and storing the results.
The packet collection device 200 is further provided with packet collection units 210a and 210b in addition to the configuration of the counting device 100 described in FIG. 1 of the first embodiment. In FIG. 7, the packet collection unit 210 is connected to the counting unit 110 and the updating unit 150, respectively, but these may be shared.

The packet collection units 210a and 210b are connected to the network, collect communication packets, extract information to be counted, for example, transmission / reception addresses of packets, and output them to the counting unit 110 and the updating unit 150, respectively.
The first N transmission / reception addresses are collected by the packet collection unit 210a, and thereafter collected by the packet collection unit 210b.
Since the processing after collecting the packets is the same as in the first embodiment, the description thereof is omitted.

Since there are an enormous number of transmission / reception addresses of communication packets on the network, a large memory capacity is required when these addresses are counted.
Therefore, by counting the transmission / reception addresses using the configuration of the counting device 100 described in the first embodiment, the transmission / reception addresses can be efficiently counted with a small memory capacity.

Embodiment 3 FIG.
In the first and second embodiments described above, the method of counting the same value appearing in the data series, for example, the same transmission / reception address, has been described. However, the object to be counted is not necessarily limited to the same data. The number of “equivalent” values may be counted as long as they meet the purpose.

  For example, in the packet collection device 200 according to the second embodiment, when it is desired to count packets transmitted / received with respect to the same network address, when the counting unit 110 performs the counting, an address that is the same value by multiplying by the subnet mask May be counted as the same value.

2 is a functional block diagram of a counting device 100 according to Embodiment 1. FIG. The division division of the data series which the counting part 110 counts is demonstrated. A counting process executed by the counting unit 110 will be described. An example of storing the count result of the counting unit 110 in the storage unit 120 is shown. The processing procedure of the discard processing unit 130 will be described. This is a procedure for using data stored in the threshold position table. 6 is a functional block diagram of a packet collection device 200 according to Embodiment 2. FIG. The example of storing the count result in a prior art is shown.

Explanation of symbols

  100 counting device, 110 counting unit, 120 storage unit, 130 discard processing unit, 140 infrequent data deleting unit, 150 updating unit, 200 packet collecting device, 210a to 210b packet collecting unit.

Claims (13)

A method of grouping equivalents of input data series and counting the appearance frequency of the equivalent data for each group,
A storage device having a storage area for storing a predetermined number of input data series is provided, and
Dividing the storage area into a number of intervals equal to the allowable counting error;
A counting step of grouping equivalent data while inputting the data series in the storage area and counting the appearance frequency of the equivalent data for each group;
When the number of the data series input to the storage area reaches the maximum size of the storage area, a discarding step for discarding all or part of the counting result counted in the initial stage;
A low-frequency data deletion step of deleting each of the groups with a low frequency of appearance;
An update step of updating the counting result by additionally inputting the number corresponding to one section of the data series;
Have
In the counting step,
A set of the counting result of each group and the allowable counting error of the group is stored in the storage device,
In the discarding step,
The count error of the group that discards the count result is updated with the count result of the group before the discard, so that the count error before and after the discard is within the range of the allowable count error. Method. In the counting step,
The count result up to the previous section of the section at the time when the appearance frequency count result exceeds the allowable count error is stored in the storage device as the allowable count error update value of the group for each group. Every
In the discarding step,
The counting method according to claim 1, wherein an allowable counting error of a group for discarding a counting result is updated with the updated allowable counting error value. In the discarding step,
When updating the allowable count error of the group for discarding the count result with the allowable count error update value,
The counting method according to claim 2, wherein a count result of the appearance frequency of the group is subtracted by the allowable count error update value. Repeating the discarding step, the infrequent data deleting step, and the updating step until the data series is completed,
In the counting step,
The number of the section at the time when the appearance frequency count result exceeds the allowable count error is stored in the storage device,
In the discarding step,
When updating the allowable count error of the group for discarding the count result with the allowable count error update value,
4. The counting according to claim 2, wherein the section number stored in the storage device in the counting step is stored in the storage device in combination with a counting result of the group. Method. Every time the update step is executed,
1 is subtracted from the number of the section stored in the storage device in the discarding step,
The counting method according to claim 4, wherein the discarding step is executed only when the number is 0. In the low frequency data deletion step,
Find the sum of the counting results for each group and the allowable counting error for that group,
The counting method according to any one of claims 1 to 5, wherein when the sum is equal to or less than the number of the sections, the counting result of the group is deleted.   A counting program for causing a computer to execute the counting method according to any one of claims 1 to 6. An apparatus that groups equivalents of input data series and counts the appearance frequency of the equivalent data for each group,
A storage device having a storage area for storing a predetermined number of input data series;
A counting unit that groups equivalent data while inputting the data series in the storage area and counts the appearance frequency of the equivalent data for each group;
When the number of the data series input to the storage area reaches the maximum size of the storage area, a discarding unit that discards all or part of the counting result counted in the initial stage;
A low-frequency data deletion unit that deletes each of the groups with a low appearance frequency;
An update unit that updates the counting result by additionally inputting the number corresponding to one section of the data series;
With
The storage area is divided into a number of intervals equal to the allowable counting error,
The counting unit is
A set of the counting result of each group and the allowable counting error of the group is stored in the storage device;
The discarding unit
The count error of the group that discards the count result is updated with the count result of the group before the discard, so that the count error before and after the discard is within the range of the allowable count error. apparatus. The counting unit is
The count results up to the previous section of the section at the time when the appearance frequency count result exceeds the allowable count error are stored in the storage device as the allowable count error update value of the group for each group,
The discarding unit
The counting device according to claim 8, wherein an allowable counting error of a group that discards the counting result is updated with the allowable counting error update value. The discarding unit
When updating the allowable count error of the group for discarding the count result with the allowable count error update value,
The counting device according to claim 9, wherein a count result of the appearance frequency of the group is subtracted by the allowable count error update value. Each process of the discard unit, the low frequency data deletion unit, and the update unit is repeated until the data series ends,
The counting unit is
The storage device stores the number of the section at the time when the appearance frequency count result exceeds the allowable count error,
The discarding unit
When updating the allowable count error of the group for discarding the count result with the allowable count error update value,
11. The counting device according to claim 9, wherein the number of the section stored in the storage device by the counting unit is stored in the storage device in combination with the count result of the group. The update unit
Each time the processing of the updating unit is executed, the discarding unit stores 1 in the section number stored in the storage device,
The discarding unit
The counting device according to claim 11, wherein the processing of the discarding unit is executed only when the number is 0. The infrequent data deletion unit is
Find the sum of the counting results for each group and the allowable counting error for that group,
The counting device according to any one of claims 8 to 12, wherein when the sum is equal to or less than the number of the sections, the counting result of the group is deleted.

Images