JP6229577B2 - Cache storage program, information processing apparatus, and cache storage method - Google Patents

Description

  The present invention relates to a cache storage program and the like.

  With the globalization of data and the use of cloud and mobile, the demand for faster communication is increasing. In response to the demand for high-speed communication, the communication device reduces the amount of data to be transferred by optimizing the data to be transferred when transferring the data. That is, the communication apparatus removes data transferred in the past from data to be transferred as duplicate data, and reduces the amount of data to be transferred.

  Here, a conventional duplicate data removal technique will be described with reference to FIG. FIG. 11 is a diagram showing a conventional duplicate data removal technique. As illustrated in FIG. 11, the communication device on the transmission side divides input data into variable-length blocks called chunks, and performs duplication determination with past input data stored in the cache memory in units of chunks. Here, a chunk refers to a variable length block obtained by dividing data on a boundary. Then, if the data corresponding to the chunk is not in the cache memory due to the duplication determination, the transmitting communication device saves the data (raw data) in the cache memory and transmits the raw data to the receiving side via the network. Transfer to device. On the other hand, if there is data corresponding to the chunk in the cache memory by duplication determination, the transmitting communication device transfers only the data index (ID: identification) to the receiving communication device via the network.

  When receiving the raw data, the communication device on the receiving side stores the received raw data in the cache memory and uses the raw data as received data. On the other hand, when receiving the data index, the communication device on the receiving side restores the data using the cache memory, and uses the restored data as received data.

  When the cache memory of the communication device on the transmission side becomes full, the data stored in the cache memory is deleted in order from the oldest unused data (LRU: Least Recently Used).

  Further, a technique for storing data in a cache memory is disclosed. In such a technique, a technique is disclosed in which a proxy server having a plurality of caches determines a cache for storing the data according to the size of the data transmitted from the web server, and stores the data in the determined cache. (For example, refer to Patent Document 1).

JP 2001-256098 A

  However, there is a problem that the communication device cannot efficiently hit the cache memory. For example, the communication device stores the data in the cache memory without considering the duplication characteristics of the data to be transferred, and thus cannot efficiently hit the cache memory. Here, an example of data duplication characteristics will be described with reference to FIGS. 12A and 12B.

  FIG. 12A is a diagram illustrating an example of duplicate data. FIG. 12A shows the flow of data to be transferred, and X is duplicate data. In the transmission data shown in FIG. 12A, the continuous length of the duplicate data X is short, but the duplicate data X occurs frequently. Such a characteristic of duplication of data to be transferred is called FSC (frequent and short chunk series).

  FIG. 12B is a diagram illustrating another example of duplicate data. FIG. 12B shows the flow of data to be transferred, and X is duplicate data. In the transmission data shown in FIG. 12B, the continuous length of the duplicate data X is long, but the duplicate data X occurs rarely. Such a characteristic of duplication of data to be transferred is called RLC (rare and long chunk series).

  In this way, the communication device stores data in the cache memory without considering the duplication characteristics of the data to be transferred, even though the data has duplication characteristics. Can not be hit.

  The above-mentioned problem is not only caused in the case of a communication apparatus that transfers data, but also occurs in the case of an information processing apparatus that processes data.

  In one aspect, an object is to efficiently hit a cache memory.

  The cache storage program disclosed in the present application stores, in a computer, when data to be processed is new, stores the data to be processed in a first cache memory, and stores each data stored in the first cache memory. In addition, the number of times of duplication that overlaps with the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as duplication history, and when the new data is processed, When the number of data stored in one cache memory is the upper limit, a predetermined deletion target data is extracted, and the deletion target data managed in the duplication history is duplicated and the continuous duplication length Then, a process of moving the data to be deleted to one of the second cache memory and the third cache memory is executed.

  According to one aspect of the cache storage program disclosed in the present application, it is possible to efficiently hit the cache memory.

FIG. 1 is a functional block diagram illustrating the configuration of the communication apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of cache storage processing of the communication apparatus according to the first embodiment. FIG. 3 is a diagram illustrating a data structure of input data and data to be transmitted. FIG. 4 is a diagram illustrating an example of a data structure of duplication management information according to the first embodiment. FIG. 5A is a diagram (1) illustrating the flowchart of the cache storage process according to the first embodiment. FIG. 5B is a diagram (2) illustrating the flowchart of the cache saving process according to the first embodiment. FIG. 5C is a diagram (3) illustrating a flowchart of the cache storage process according to the first embodiment. FIG. 6 is a functional block diagram illustrating the configuration of the communication apparatus according to the second embodiment. FIG. 7 is a diagram illustrating an example of cache storage processing of the communication apparatus according to the second embodiment. FIG. 8 is a diagram illustrating an example of a data structure of duplication management information according to the second embodiment. FIG. 9A is a diagram (1) illustrating the flowchart of the cache storage process according to the second embodiment. FIG. 9B is a diagram (2) illustrating the flowchart of the cache saving process according to the second embodiment. FIG. 9C is a diagram (3) illustrating the flowchart of the cache storage process according to the second embodiment. FIG. 9D is a diagram (4) illustrating the flowchart of the cache saving process according to the second embodiment. FIG. 10 is a diagram illustrating an example of a computer that executes a cache storage program. FIG. 11 is a diagram showing a conventional duplicate data removal technique. FIG. 12A is a diagram illustrating an example of duplicate data. FIG. 12B is a diagram illustrating another example of duplicate data.

  Embodiments of a cache storage program, an information processing apparatus, and a cache storage method disclosed in the present application will be described below in detail with reference to the drawings. The present invention is not limited to the embodiments.

[Configuration of communication device]
FIG. 1 is a functional block diagram illustrating the configuration of the communication apparatus according to the first embodiment. The communication device 1 is a communication device that transmits data transmitted and received between communication devices. The communication device 1 hierarchizes a cache storage area for storing data transmitted in the past, and moves data to a predetermined lower cache at a timing when the upper cache is full according to the data duplication characteristics.

  The communication device 1 includes a storage unit 10 and a control unit 20. The storage unit 10 corresponds to a storage device such as a nonvolatile semiconductor memory element such as a flash memory or a FRAM (registered trademark) (Ferroelectric Random Access Memory). The storage unit 10 includes a cache storage area that serves as a cache memory. The cache storage area includes a memory 11, a disk 12, and an SSD (Solid State Drive) 13, and the disk 12 and the SSD 13 are arranged under the memory 11.

Here, an example of the cache storage process of the communication apparatus 1 according to the first embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of cache storage processing of the communication apparatus according to the first embodiment. As illustrated in FIG. 2, the communication device 1 divides input data into variable-length blocks called chunks, and performs duplication determination with past input data stored in the memory 11 in units of chunks.
If there is data corresponding to the chunk in the memory 11 as a result of the duplication determination, the communication device 1 sends only the data index (ID: identification) to the receiving communication device via the network in order to remove the duplicate data. Forward.

  On the other hand, if there is no data corresponding to the chunk in the memory 11 due to the duplication determination, the communication device 1 stores the data (raw data) in the memory 11 and sends the raw data to the receiving communication device via the network. Send. When the memory 11 becomes full when the data is stored in the memory 11, the communication device 1 among the chunks stored in the memory 11 is the oldest unused chunk (LRU: Least Recently Used). To extract. Then, the communication device 1 sets the extracted chunk as a deletion target.

  If the communication device 1 determines that the data to be deleted has duplication rarely and the duplication continues for a long time, the communication device 1 moves the data to be deleted from the memory 11 in the cache storage area to the disk 12. That is, when the data duplication characteristic is RLC (rare and long chunk series), the communication device 1 moves the data to the low-speed disk 12 of the hierarchical cache.

  If the communication device 1 determines that the data to be deleted frequently overlaps and continues for a short time, the communication device 1 moves the data to be deleted from the memory 11 in the cache storage area to the SSD 13. That is, when the data duplication characteristic is FSC (frequent and short chunk series), the communication device 1 moves the data to the high-speed SSD 13 of the hierarchical cache.

  As a result, the communication device 1 stores data in the cache storage area in consideration of data duplication characteristics, and thus it is possible to efficiently hit the cache storage area.

  FIG. 3 is a diagram illustrating a data structure of input data and data to be transmitted. Note that A, B, C, and D shown in FIG. 3 each represent a chunk. That is, the input data is divided into A, B, C, B, and D chunks. As shown in FIG. 3, the appearance position of each chunk of input data continues. The “appearance position” here refers to a kind of address that increases each time a new chunk appears and does not increase when the chunks overlap. The data amount of the input data is a total number 5200 of chunk A 1200, chunk B 1000, chunk C 1100, chunk B 1000, and chunk D 900. However, since chunk B is duplicated, the amount of data to be transmitted is 1200 for chunk A, 1000 for chunk B, 1100 for chunk C, and a total of 4200 for 900 for chunk D. That is, since the chunk B is duplicated, the data amount is 0 except for the ID, and the appearance position is 0.

  Returning to FIG. 1, the memory 11 stores duplication management information 111. The duplication management information 111 manages the duplication history with the input data for the already stored chunks. That is, the duplication management information 111 manages, as duplication history, the data length and the number of times of duplication with the input data for each of a plurality of chunks obtained by dividing already transmitted data. The data structure of the duplication management information 111 will be described later.

  The control unit 20 has an internal memory for storing programs defining various processing procedures and control data, and executes various processes using these. And the control part 20 respond | corresponds to the electronic circuit of integrated circuits, such as ASIC (Application Specific Integrated Circuit) and FPGA (Field Programmable Gate Array). Alternatively, the control unit 20 corresponds to an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Furthermore, the control unit 20 includes a chunk division unit 21, a hash calculation unit 22, a duplication determination unit 23, a new registration unit 24, a duplication management unit 25, a cache migration unit 26, and a transmission unit 27.

  The chunk division unit 21 divides input data into a plurality of chunks. Here, a chunk means a variable length block obtained by distinguishing data by boundary. A method of dividing input data into a plurality of chunks is executed as follows.

For example, the chunk division unit 21 scans input data with a fixed window size. Further, the chunk division unit 21 calculates a hash value of the window data by a method called “Rabin” fingerprint (abbreviated as “fp”). As an example, assuming that the window size is n and the data in the window is w ₀ , w ₁ ,..., W _n−1 , fp is expressed by the following equation (1). In addition, p shown by Formula (1) is a comparatively big prime number.

Further, the chunk division unit 21 calculates fp extended by 1 byte by using the result of Expression (1) and Expression (2).

Further, the chunk division unit 21 searches for a fp in which m bits in the rear (or front) match a predetermined bit pattern among a plurality of fps obtained by calculation. The boundary of data is obtained by fp that can be searched. Since the probability that the m bits of fp match is a probability of 1/2 ^m , for example, if m is “10”, a data boundary is obtained at a rate of once every 2 ¹⁰ (1 KB) on average. The chunk division unit 21 divides the found fp data as one chunk. Note that the method of dividing the input data into a plurality of chunks is not limited to this, and any known technique may be used.

  The hash calculator 22 calculates a hash value for the chunk divided by the chunk divider 21. For example, the chunk calculation unit 22 sequentially selects the chunks divided by the chunk division unit 21. The chunk calculation unit 22 calculates a hash value for the selected chunk using SHA1 (Secure Hash Algorithm 1).

  The duplication determination unit 23 determines whether or not the selected chunk overlaps any one of the plurality of chunks stored in the memory 11. For example, the duplication determination unit 23 uses the duplication management information 111 to determine whether the calculated hash value of the selected chunk matches the hash value of the chunk already stored in the memory 11.

  When it is determined that the selected chunk does not overlap any of the plurality of chunks stored in the memory 11, the new registration unit 24 newly registers the selected chunk in the memory 11. For example, when the new registration unit 24 determines that the calculated hash value of the selected chunk does not match any hash value of the chunk already stored in the memory 11 by the duplication determination unit 23, Save in the memory 11. Further, the new registration unit 24 registers the information of the selected chunk in the duplication management information 111.

  Here, the data structure of the duplication management information 111 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a data structure of duplication management information according to the first embodiment. As shown in FIG. 4, the duplication management information 111 stores the appearance position 111b, the length 111c, the number of appearances 111d, and the continuation 111e in association with the hash value (SHA1) 111a. The hash value (SHA1) 111a indicates a hash value calculated by the SHA1 of the chunk. The appearance position 111b indicates the appearance position of the chunk. The length 111c indicates the length of the chunk. The appearance count 111d indicates the number of times the chunk has already appeared. The continuation 111e indicates the number of chunks in which duplication continues in a plurality of chunks into which input data is divided. The continuation 111e is set to a portion corresponding to the first chunk of the input data. Note that the appearance count 111d and the continuation 111e are set by the duplication management unit 25 described later. Further, the appearance count 111d and the continuation 111e correspond to the continuation overlap length and the overlap count, respectively, as an example.

  As an example, information of each chunk when input data including chunks a, b, and c is input and subsequently input data including chunks a and b is input is stored. The information whose hash value (SHA1) 111a is “aaa” is the information of chunk a. The information whose hash value (SHA1) 111a is “bbb” is the information of chunk b. The information whose hash value (SHA1) 111a is “ccc” is the information of chunk c. Chunk a is the head of the input data. When the hash value (SHA1) 111a is “aaa”, “0” is stored as the appearance position 111b, “1200” is stored as the length 111c, and “2” is stored as the appearance count 111d. Then, “2” is stored as the continuation 111e. This is because the number of chunks in which duplication continues is two, a and b.

  Returning to FIG. 1, the new registration unit 24 adds the hash value calculated by the hash calculation unit 22 as the hash value (SHA1) 111a and the data amount of the new chunk as the appearance position 111b for the selected chunk. Set the obtained offset value. The new registration unit 24 sets the length of the chunk as the length 111c, “1” as the number of appearances 111d, and “0” as the continuation 111e.

  The duplication management unit 25 manages duplication of the selected chunk when it is determined that the selected chunk overlaps any one of the plurality of chunks stored in the memory 11. For example, when the duplication management unit 25 determines that the calculated hash value of the selected chunk matches the hash value of one of the chunks already stored in the memory 11, Process. If the leading chunk of the input data is not recognized, the duplication management unit 25 recognizes the selected chunk as the leading chunk of the input data, and adds 1 to the continuation 111e corresponding to the selected chunk. The duplication management unit 25 adds 1 to the continuation 111e corresponding to the first chunk if the first chunk of the input data has already been recognized and the selected chunk continues with the previously duplicated chunk. To do. When the first chunk of the input data is already recognized, the duplication management unit 25 determines that the selected chunk is the first chunk of the input data if the selected chunk does not continue with the most recently duplicated chunk. Recognize and add 1 to the continuation 111e corresponding to the selected chunk. Further, the duplication management unit 25 adds 1 to the appearance count 111d of the selected chunk. In addition, the duplication management unit 25 sets information on the selected chunk at the head of the duplication management information 111.

  When registering a new chunk in the memory 11, the cache moving unit 26 extracts an LRU chunk from the chunks stored in the memory 11 when the number of chunks stored in the memory 11 is the upper limit. The extracted chunk is a deletion target. The cache moving unit 26 moves the data including the deletion target chunk to one of the caches of the disk 12 and the SSD 13 according to the predicted continuation length at the time of duplication in the deletion target chunk and the number of occurrences of the duplication. For example, the cache transfer unit 26 sets the last chunk of the duplication management information 111 as a deletion target. In addition, when the continuation 111e of the chunk to be deleted is larger than a predetermined continuation threshold, the cache transfer unit 26 moves the chunk data for the continuation 111e that continues from the chunk to the disk 12. That is, when the chunk duplication characteristic is RLC (rare and long chunk series), the cache mover 26 moves the corresponding data to the low-speed disk 12 of the hierarchical cache. In addition, when the deletion target chunk continuation 111e is equal to or smaller than a predetermined continuation threshold value, the cache transfer unit 26 determines that the deletion target chunk is greater than a predetermined appearance frequency threshold value. The included data is moved to the SSD 13. In other words, when the characteristic of chunk overlap is FSC (frequent and short chunk series), the cache mover 26 moves the corresponding data to the high-speed SSD 13 of the hierarchical cache. Further, the cache transfer unit 26 deletes the moved chunk from the memory 11 and deletes the moved chunk information from the duplication management information 111.

  If the selected chunk is duplicated, the transmitting unit 27 transmits only the ID of the chunk whose contents are duplicated. Moreover, the transmission part 27 transmits the data (raw data) corresponding to a chunk, when it does not overlap about the selected chunk.

[Cache saving processing procedure]
Next, the procedure of the cache storage process according to the first embodiment will be described with reference to FIGS. 5A to 5C. 5A to 5C are flowcharts illustrating the cache storage process according to the first embodiment. Note that the first chunk being duplicated is referred to as “duplicate first chunk” and is represented by “HEADCHUNK”. The duplicate chunk immediately before the duplicate chunk is called “immediate duplicate chunk”, and is represented by “PRECHUNK”. The current appearance position is referred to as “current appearance position” and is represented by “OFFSET”. The number of chunks stored in the memory 11 is represented by “datanum”. HEADCHUNK, PRECHUNK, OFFSET, and dataum are local areas temporarily stored in the storage unit 10.

  As shown in FIG. 5A, the chunk division unit 21 sets null to HEADCHUNK and PRECHUNK, and sets 0 to OFFSET and dataum (step S11). The chunk division unit 21 waits for data reception (step S12).

  The chunk division unit 21 determines whether the data is complete (step S13). When the data is finished (step S13; Yes), the chunk division unit 21 finishes the cache storage process.

  On the other hand, when the data is not finished (step S13; No), the chunk division unit 21 receives the data (input data) (step S14). The chunk division unit 21 divides the received input data into a plurality of chunks (step S15). Then, the chunk dividing unit 21 selects one chunk from the divided chunks (step S16).

  Subsequently, the control unit 20 performs a cache storage process on the selected chunk (step S17). Note that the flowchart of the cache storage process will be described with reference to FIGS. 5B and 5C. And the control part 20 determines whether the process in all the chunks was complete | finished (step S18). If the processing has not been completed for all the chunks (step S18; No), the control unit 20 proceeds to step S16 to select the next chunk.

  On the other hand, when the processing in all the chunks is completed (step S18; Yes), the control unit 20 proceeds to step S12 so as to wait for reception of the next data.

  As shown in FIG. 5B, the hash calculator 22 calculates a hash value for the selected chunk using SHA1 (step S21). The hash calculation unit 22 sets the calculated hash value in the SHAVAL that is the local area. And the duplication determination part 23 searches for duplication of the selected chunk (step S22).

  And the duplication determination part 23 determines whether the selected chunk is new (step S23). For example, the duplication determination unit 23 determines whether the hash value of the selected chunk does not match any hash value (SHA1) 111a stored in the duplication management information 111.

  If the selected chunk is new (step S23; Yes), the new registration unit 24 registers the new chunk information in the duplication management information 111 (step S24). For example, the value set in SHAVAL is set in the hash value (SHA1) 111a. A value set to OFFSET is set at the appearance position 111b. The chunk length is set to the length 111c. “1” is set in the appearance count 111d. “0” is set in the continuation 111e.

  Then, the new registration unit 24 adjusts HEADCHUNK (overlapping head chunk), OFFSET (current appearance position), and dataum (number of chunks stored in the memory 11), which are local areas (step S25). For example, null is set in HEADCHUNK. A value obtained by adding the length of the selected chunk to OFFSET is set. A value obtained by adding 1 to the selected chunk is set to dataum.

  Then, the new registration unit 24 sets the new chunk information at the head of the duplication management information 111 (step S26). This is because this new chunk has been used recently. Then, the new registration unit 24 proceeds to step S41 in order to perform the cache transfer process in the cache storage process. Note that a flowchart of the cache transfer process in the cache storage process will be described with reference to FIG. 5C.

  If the selected chunk is not new in step S23 (step S23; No), the duplication management unit 25 acquires information on the selected chunk (hereinafter, registered chunk) registered in the duplication management information 111. (Step S27). Subsequently, the duplication management unit 25 determines whether or not HEADCHUNK (duplication head chunk) is null (step S28). That is, the duplication management unit 25 determines whether or not the leading chunk of the input data is recognized.

  If HEADCHUNK (duplicate head chunk) is null (step S28; Yes), the duplication management unit 25 sets the hash value of the registered chunk in HEADCHUNK (step S29). That is, the duplication management unit 25 recognizes the registered chunk as the first chunk of duplication. And the duplication management part 25 transfers to step S32.

  On the other hand, when HEADCHUNK (duplicate head chunk) is not null (step S28; No), the duplication management unit 25 adds the appearance position and length of PRECHNK (preceding duplicate chunk) to the registered chunk. It is determined whether it is the appearance position of (step S30). That is, the duplication management unit 25 determines whether the registered chunk is continued with the chunk that was duplicated immediately before.

  When the value obtained by adding the appearance position and the length of PRECHUNK (preceding duplicate chunk) is not the appearance position of the registered chunk (step S30; No), the duplication management unit 25 hashes the chunks registered in HEADCHUNK. A value is set (step S31). That is, since the registered chunk does not continue with the previously duplicated chunk, the duplication management unit 25 recognizes the registered chunk as a duplicated first chunk. And the duplication management part 25 transfers to step S32.

  When the value obtained by adding the appearance position and the length of PRECHNK (preceding duplicate chunk) is the appearance position of the registered chunk (step S30; Yes), the duplication management unit 25 determines that the first chunk of duplication is Since it has already been recognized, the process proceeds to step S32.

  In step S32, the duplication management unit 25 adds 1 to the HEADCHUNK continuation 111e (step S32). That is, the duplication management unit 25 adds 1 to the continuation 111e corresponding to the first chunk of duplication. And the duplication management part 25 adds 1 to the appearance count 111d of the registered chunk (step S33). The duplication management unit 25 sets the hash value of the registered chunk in PRECHUNK (previous duplication chunk) (step S34).

  And the duplication management part 25 sets the information of the registered chunk to the head of duplication management information 111 (step S35). This is because the registered chunk was used the latest. Then, the duplication management unit 25 returns to step S18 to select the next chunk.

  As shown in FIG. 5C, the cache transfer unit 26 executes a cache transfer process as follows. The cache migration unit 26 determines whether or not the dataum (number of chunks stored in the memory 11) exceeds the upper limit value (step S41). If the dataumum does not exceed the upper limit (step S41; No), the cache transfer unit 26 ends the cache transfer process and returns to step S18.

  On the other hand, when the dataumum exceeds the upper limit value (step S41; Yes), the cache transfer unit 26 sets the last chunk of the duplication management information 111 as a deletion chunk (step S42). That is, the cache transfer unit 26 sets the last chunk of the duplication management information 111 as a deletion target.

  Then, the cache migration unit 26 determines whether or not the deletion chunk continuation 111e is larger than the continuation threshold (step S43). When the deletion chunk continuation 111e is larger than the continuation threshold (step S43; Yes), the cache transfer unit 26 saves the data including the deletion chunk to the disk 12 (step S44). For example, the cache transfer unit 26 moves the chunk data for 111e continuing from the deleted chunk to the disk 12. That is, when the chunk duplication characteristic is RLC (rare and long chunk series), the cache mover 26 moves the corresponding data to the low-speed disk 12 of the hierarchical cache. Then, the cache transfer unit 26 proceeds to Step S48.

  When the deletion chunk continuation 111e is equal to or less than the continuation threshold (step S43; No), the cache migration unit 26 determines whether or not the deletion chunk appearance count 111d is larger than the appearance count threshold (step S45). When the appearance count 111d of the deleted chunk is larger than the appearance count threshold (step S45; Yes), the cache transfer unit 26 saves the data including the deleted chunk to the SSD 13 (step S46). For example, the cache moving unit 26 moves the chunk data for 111e continuing from the deleted chunk to the SSD 13. In other words, when the characteristic of chunk overlap is FSC (frequent and short chunk series), the cache mover 26 moves the corresponding data to the high-speed SSD 13 of the hierarchical cache. Then, the cache transfer unit 26 proceeds to Step S48.

  On the other hand, when the deleted chunk appearance count 111d is equal to or smaller than the appearance count threshold (step S45; No), the cache transfer unit 26 discards the deleted chunk (step S47). Then, the cache transfer unit 26 proceeds to Step S48.

  In step S48, the cache transfer unit 26 subtracts 1 from dataum (number of chunks stored in the memory 11) (step S48). Then, the cache transfer unit 26 ends the cache transfer process and returns to step S18.

[Effect of Example 1]
According to the first embodiment, the communication device 1 stores the processing target data in the memory 11 when the processing target data is new. For each piece of data stored in the memory 11, the communication device 1 uses, as the duplication history, the number of times of duplication that overlaps with the data to be processed and the continuous duplication length that indicates the length of duplication that continues when the data to be processed overlaps to manage. When processing new data, the communication device 1 extracts predetermined data to be deleted when the number of data stored in the memory 11 is the upper limit. Then, the communication device 1 moves the data to be deleted to one of the caches of the disk 12 and the SSD 13. According to such a configuration, when the number of data in the memory 11 is the upper limit, the communication device 1 moves the data to be deleted to one of the caches of the disk 12 and the SSD 13. It becomes possible to hit the cache efficiently.

  Further, according to the first embodiment, the communication device 1 moves the deletion target data to the low-speed disk 12 among the disk 12 and the SSD 13 when the continuous overlap length of the deletion target data is larger than the first threshold. To do. According to such a configuration, the communication device 1 can store the deletion target data in an optimum cache in accordance with the duplication characteristic (RLC) of the deletion target data.

  Further, according to the first embodiment, the communication device 1 determines that the continuous duplication length of the data to be deleted is equal to or smaller than the first threshold and the duplication count of the data to be deleted is larger than the second threshold. The data to be deleted is moved to the high-speed SSD 13 out of the disk 12 and the SSD 13. According to such a configuration, the communication device 1 can store the deletion target data in an optimal cache in accordance with the duplication characteristic (FSC) of the deletion target data.

  Further, according to the first embodiment, the communication device 1 has a case where the continuous duplication length of the data to be deleted is equal to or smaller than the first threshold and the number of times of duplication of the data to be deleted is equal to or smaller than the second threshold. In addition, the data to be deleted is discarded. According to this configuration, the communication device 1 can increase the capacity for storing important data by discarding unnecessary data from the cache.

  By the way, in the first embodiment, the communication apparatus 1 has described the case where the cache storage area for storing data transmitted in the past is hierarchized. That is, when the data stored in the memory 11 is full, the communication device 1 moves the predetermined data to the hierarchical cache according to the duplication characteristics of the predetermined data stored in the memory 11. To do. However, the communication apparatus 1 is not limited to this, and when the capacity of the memory 11 that can be used as a cache is small, the communication apparatus 1 may store the data transmitted in the past in the memory 11 while thinning out the data.

  Therefore, in the second embodiment, a case will be described in which the communication device 1 further stores data transmitted in the past in the memory 11 when the capacity of the memory 11 usable as a cache is small.

[Configuration of communication device]
FIG. 6 is a functional block diagram illustrating the configuration of the communication apparatus according to the second embodiment. In addition, about the same structure as the communication apparatus 1 shown in FIG. 1, the same code | symbol is shown, and the description of the overlapping structure and operation | movement is abbreviate | omitted. The difference between the first embodiment and the second embodiment is that the overlap determination unit 23 is changed to the overlap determination unit 31. The difference between the first embodiment and the second embodiment is that the new registration unit 24 is changed to the sampling chunk registration unit 32. The difference between the first embodiment and the second embodiment is that a disk 14 is added. The disk 14 stores duplication management information 211.

  Here, an example of the cache storage process of the communication device 1 according to the second embodiment will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of cache storage processing of the communication apparatus according to the second embodiment. As shown in FIG. 7, the communication device 1 divides the input data A into variable length blocks called chunks. Then, the communication device 1 stores the divided chunks in the memory 11 and the disk 14 according to the usage ratio of the memory 11. For example, when the usage ratio of the memory 11 is ½, the communication device 1 increments the counter by 1 each time a new chunk appears, and divides this counter by 2, which is the denominator of the usage ratio of the memory 11. Change the chunk storage location using the remainder. As an example, the storage destination of the chunk with a remainder of 0 is the memory 11 (or disk 14), and the storage destination of the chunk with a remainder of 1 is the disk 14 (or memory 11).

  In FIG. 7, the communication device 1 stores the even-numbered chunk in the memory 11 that is the primary cache for each divided chunk of the input data A at the first cache of the input data A, and stores the odd-numbered chunk. Save to the disk 14 which is a secondary cache. Although the case where the usage ratio of the memory 11 is ½ has been described, the present invention is not limited to this, and the usage ratio of the memory 11 may be determined according to the capacity of the memory 11. As an example, when the usage ratio of the memory 11 is 3/10, the storage destination of the chunk whose remainder is 0 to 2 divided by 10 is the memory 11, and the chunk whose remainder is 3 to 9 is divided by 10 The storage destination may be the disk 14.

  When the memory 11 becomes full when data is stored in the memory 11, the communication device 1 stores the oldest unused chunk (LRU: Least Recently) among the chunks stored in the memory 11. Used) is extracted. Then, the communication device 1 sets the extracted chunk as a deletion target.

  Here, it is assumed that the chunk to be deleted is the chunk c1 of the input data A. Then, the communication device 1 moves the continuation chunk data c1 to c7 from the deletion target chunk c1 to the disk 12 or the SSD 13. When the number of chunks for the continuation is larger than the continuation threshold, the communication device 1 moves the data to the low-speed disk 12 of the hierarchical cache. The communication device 1 deletes the data of the deletion target chunk c1 from the memory 11.

  Thereafter, in the second cache of the input data A, the communication device 1 makes a cache hit because all the chunks c1 to c7 of the input data A are stored in the cache storage area. And the communication apparatus 1 can remove and transmit the duplicate chunks c1-c7 of the duplicate input data A.

  As a result, the communication device 1 stores the data using the disk 14 of the secondary cache together even when the capacity of the memory 11 of the primary cache is small. Therefore, the communication device 1 can realize data transmission deduplication without losing a part of data whose data duplication characteristic is RLC (rare and long chunk series).

  Returning to FIG. 6, the duplication determination unit 31 determines whether or not the selected chunk overlaps any one of the plurality of chunks stored in the memory 11. For example, the duplication determination unit 31 uses the duplication management information 111 to determine whether the calculated hash value of the selected chunk matches the hash value of the chunk already stored in the memory 11.

  In addition, the duplication determination unit 31 determines that the selected chunk has been found when it overlaps with any one of the plurality of chunks stored in the memory 11. Then, the duplication determination unit 31 sets the selected chunk as a chunk for thinning-out chunk search.

  In addition, when there is no duplication with any of the plurality of chunks stored in the memory 11, the duplication determination unit 31 selects the selected chunk as one of the plurality of chunks stored in the disk 14. Determine whether they overlap. For example, the duplication determination unit 31 stores the calculated hash value of the selected chunk in the disk 14 using the decimation chunk position for decimation chunk search and the duplication management information 211 stored in the duplication management information 111. It is determined whether or not it matches the hash value of the designated chunk.

  The duplication determination unit 31 determines that the selected chunk has been found when the selected chunk overlaps any one of the plurality of chunks stored in the disk 14. The duplication determination unit 31 determines that the selected chunk has not been found when the selected chunk does not overlap with any of the plurality of chunks stored in the disk 14.

  Here, the data structures of the duplication management information 111 stored in the memory 11 and the duplication management information 211 stored in the disk 14 will be described with reference to FIG. FIG. 8 is a diagram illustrating an example of a data structure of duplication management information according to the second embodiment.

  The upper part of FIG. 8 is a diagram illustrating an example of the data structure of the duplication management information 111 stored in the memory 11. Regarding the duplication management information 111 stored in the memory 11, the same components as those in the duplication management information 111 shown in FIG. 8 differs from the duplication management information 111 shown in FIG. 4 in that a thinned chunk position 111f is added. The lower part of FIG. 8 is a diagram showing an example of the data structure of the duplication management information 211 stored in the disk 14. Since the duplication management information 111 stored in the disk 14 shown in the lower diagram of FIG. 8 has the same configuration as that of the duplication management information 111 shown in FIG. 4, description of the duplicate configuration is omitted.

  The thinned chunk position 111f indicates the position stored in the duplication management information 211 of the disk 14 regarding the thinned chunk information. That is, the thinned chunk position 111f indicates the position of the thinned chunk information. In FIG. 8, information on each chunk x and y included in the input data is stored. The information whose hash value (SHA1) 111a is “xxx” is the information of chunk x. The information whose hash value (SHA1) 211a is “yyy” is chunk y information. The position to the information of the chunk y is set in the thinned chunk position 111f of the chunk x. Since the hash value of the chunk x is found in the duplicate management information 111 on the memory 11, the duplication determination unit 31 sets the chunk x as a chunk for thinning-out chunk search. Since the hash value of chunk y was not found in the duplicate management information 111 on the memory 11, the duplication determination unit 31 traces the chunk information stored in the disk 14 from the thinned chunk position 111f for the thinned chunk search chunk x. . Then, the duplication determination unit 31 determines whether the calculated hash value of the chunk y matches the hash value of the duplication management information 211 on the disk 14, and finds information on the chunk y.

  Returning to FIG. 6, when the selected chunk is not found by the duplication determination unit 31, the sampling chunk registration unit 32 determines the storage destination of the selected chunk. For example, the sampling chunk registration unit 32 calculates the remainder obtained by dividing the value of the counter that is incremented by one each time a new chunk appears by the value of the denominator of the predetermined usage ratio of the memory 11. In addition, the sampling chunk registration unit 32 stores the selected chunk in the memory 11 when the remainder is 0 or more and is smaller than the predetermined numerator value of the usage ratio of the memory 11. Further, the sampling chunk registration unit 32 stores the selected chunk in the disk 14 when the remainder is equal to or greater than a predetermined numerator value of the usage ratio of the memory 11.

[Cache saving processing procedure]
Next, the procedure of the cache storage process according to the second embodiment will be described with reference to FIGS. 9A to 9D. 9A to 9D are flowcharts of the cache storage process according to the second embodiment. Regarding the flowcharts shown in FIGS. 9A and 9B, the same operations as those in the flowcharts shown in FIGS. 5A and 5B are denoted by the same reference numerals, and the description of the overlapping operations will be omitted. Note that the first chunk being duplicated is referred to as “duplicate first chunk” and is represented by “HEADCHUNK”. The duplicate chunk immediately before the duplicate chunk is called “immediate duplicate chunk”, and is represented by “PRECHUNK”. The current appearance position is referred to as “current appearance position” and is represented by “OFFSET”. The number of chunks stored in the memory 11 is represented by “datanum”. The number of new chunks is referred to as “chunk counter” and is represented by “counter”. HEADCHUNK, PRECHUNK, OFFSET, dataumum and counter are local areas temporarily stored in the storage unit 10. Further, the usage ratio of the memory 11 is assumed to be n / m (n, m: a positive integer).

  As shown in FIG. 9A, the chunk division unit 21 sets null to HEADCHUNK and PRECHUNK, and sets 0 to OFFSET, datatum, and counter (step S11A). The chunk division unit 21 waits for data reception (step S12).

  The chunk division unit 21 determines whether the data is complete (step S13). When the data is finished (step S13; Yes), the chunk division unit 21 finishes the cache storage process.

  On the other hand, when the data is not finished (step S13; No), the chunk division unit 21 receives the data (input data) (step S14). The chunk division unit 21 divides the received input data into a plurality of chunks (step S15). Then, the chunk dividing unit 21 selects one chunk from the divided chunks (step S16).

  Subsequently, the control unit 20 performs a cache storage process on the selected chunk (step S17). The flowchart of the cache storage process will be described with reference to FIGS. 9B to 9D. And the control part 20 determines whether the process in all the chunks was complete | finished (step S18). If the processing has not been completed for all the chunks (step S18; No), the control unit 20 proceeds to step S16 to select the next chunk.

  On the other hand, when the processing in all the chunks is completed (step S18; Yes), the control unit 20 proceeds to step S12 so as to wait for reception of the next data.

  As shown in FIG. 9B, the hash calculator 22 calculates a hash value for the selected chunk using SHA1 (step S21). The hash calculation unit 22 sets the calculated hash value in the SHAVAL that is the local area.

  And the duplication determination part 31 searches for duplication of the selected chunk (step S22A). A flowchart of the chunk duplication search process will be described with reference to FIG. 9D.

  And the duplication determination part 31 determines whether the selected chunk is new based on the duplication search process of a chunk (step S23). For example, the duplication determination unit 31 determines whether the selected chunk was not found by the chunk duplication search process. If the selected chunk is not found, the selected chunk is new. If the selected chunk is found, the selected chunk is not new.

  If the selected chunk is new (step S23; Yes), the sampling chunk registration unit 32 executes a sampling chunk registration process (step S24A). That is, in the sampling chunk registration process, the selected chunk is sampled and registered in the memory 11 or the disk 14 according to the sampling. A flowchart of the sampling chunk registration process will be described with reference to FIG. 9C.

  Then, the sampling chunk registration unit 32 adjusts HEADCHUNK (overlapping head chunk), OFFSET (current appearance position), dataum (number of chunks stored in the memory 11), and counter (chunk counter), which are local areas (step). S25A). For example, null is set in HEADCHUNK. A value obtained by adding the length of the selected chunk to OFFSET is set. A value obtained by adding 1 to the selected chunk is set to dataum. A value obtained by adding 1 to the selected chunk is set in the counter.

  Then, the new registration unit 24 sets the new chunk information at the head of the duplication management information 111 (step S26). This is because this new chunk has been used recently. Then, the new registration unit 24 performs a cache transfer process in the cache saving process. Since the flowchart of the cache transfer process in the cache storage process has been described with reference to FIG. 5C, the description of the overlapping operation is omitted.

  In step S23, when the selected chunk is not new (step S23; No), the duplication management unit 25 executes duplication management processing. In addition, since the flowchart (steps S27 to S35) of the duplication management process has been described with reference to FIG. 5B, description of the overlapping operation is omitted.

  As illustrated in FIG. 9C, the sampling chunk registration unit 32 performs the sampling chunk registration process as follows. The sampling chunk registration unit 32 calculates a remainder obtained by dividing the counter (chunk counter) by the denominator value m of the usage ratio of the memory 11 in order to determine the storage destination of the selected chunk (step S51).

  Then, the sampling chunk registration unit 32 determines whether or not the remainder is 0 or more and smaller than n which is the numerator value of the usage ratio of the memory 11 (step S52). If the remainder is not less than 0 and smaller than n, which is the numerator value of the usage ratio of the memory 11 (step S52; Yes), the sampling chunk registration unit 32 stores the selected chunk in the memory 11 (step S53).

  Then, the sampling chunk registration unit 32 registers the information of the selected chunk in the duplication management information 111 on the memory 11 (Step S54). For example, the value set in SHAVAL is set in the hash value (SHA1) 111a. A value set to OFFSET is set at the appearance position 111b. The chunk length is set to the length 111c. “1” is set in the appearance count 111d. “0” is set in the continuation 111e.

  Then, the sampling chunk registration unit 32 determines whether or not the remainder is a value obtained by subtracting 1 which is a numerator value of the usage ratio of the memory 11 (step S55). That is, the sampling chunk registration unit 32 determines whether or not it is the last chunk stored in the memory 11 among the plurality of chunks of the input data received this time. As an example, it is assumed that the usage ratio of the memory 11 is 2/5. It is assumed that the remainders of the divided chunks a, b, and c are 0, 1, and 2. Then, the remainder of the chunk b is 1, and the remainder of the chunk b is a value 1 obtained by subtracting 1 which is 2 that is the numerator value of the utilization ratio of the memory 11. Therefore, chunk b is the last chunk stored in memory 11 among the plurality of chunks of input data received this time.

  If the remainder is a value obtained by subtracting n, which is the numerator value of the utilization ratio of the memory 11 (step S55; Yes), the sampling chunk registration unit 32 stores the selected chunk for thinning chunk position storage. (Step S56). That is, the sampling chunk registration unit 32 sets the selected chunk as a chunk for updating the thinning position of the subsequent chunk. And the sampling chunk registration part 32 performs a sampling chunk registration process, and returns to step S25A.

  On the other hand, when the remainder is not a value obtained by subtracting 1 which is the numerator value of the usage ratio of the memory 11 (step S55; No), the sampling chunk registration unit 32 stores the selected chunk for saving the thinned chunk position. Don't chunk. And the sampling chunk registration part 32 complete | finishes a sampling chunk registration process, and returns to step S25A.

  In step S52, when the remainder is not less than 0 and not smaller than n which is the numerator value of the utilization ratio of the memory 11 (step S52; No), the sampling chunk registration unit 32 stores the selected chunk in the disk 14. (Step S57).

  Then, the sampling chunk registration unit 32 registers the selected chunk information in the duplication management information 211 on the disk 14 (step S58). For example, the value set in SHAVAL is set in the hash value (SHA1) 211a. A value set to OFFSET is set in the appearance position 211b. The chunk length is set in the length 211c. “1” is set in the appearance count 211d. In the continuation 211e, “0” is set.

  Then, the sampling chunk registration unit 32 updates the thinning position of the chunk for saving the thinned chunk position (step S59). For example, the position of the selected chunk in the duplication management information 211 is set in the thinning chunk position 111f for the chunk for saving the thinned chunk position of the duplication management information 111. And the sampling chunk registration part 32 complete | finishes a sampling chunk registration process, and returns to step S25A.

  As illustrated in FIG. 9D, the duplication determination unit 31 performs chunk duplication search processing as follows. The duplication determination unit 31 searches the memory 11 for duplication of the selected chunk (step S61). For example, the duplication determination unit 31 searches for duplication of the selected chunk using the duplication management information 111 on the memory 11 and the hash value of the selected chunk.

  And the duplication determination part 31 determines whether the selected chunk was found in the memory 11 (step S62). When the selected chunk is found (step S62; Yes), the duplication determination unit 31 sets the selected chunk as a chunk for thinning-out chunk search (step S63). Then, the duplication determination unit 31 returns to step S23 with the fact that the selected chunk has been found as a parameter.

  On the other hand, when the selected chunk is not found (step S62; No), the duplication determination unit 31 searches for a thinned-out chunk position for the thinned-out chunk search chunk (step S64). That is, the duplication judgment unit 31 searches the disc 14 for duplication of the selected chunk. For example, the duplication determination unit 31 searches for duplication of the selected chunk using the decimation chunk position 111f for the decimation chunk search chunk stored in the duplication management information 111 and the duplication management information 211 on the disk 14. .

  Then, the duplication determination unit 31 determines whether or not the selected chunk has been found on the disk 14 (step S65). When the selected chunk is not found (step S65; No), the duplication determination unit 31 returns to step S23 with the parameter that the selected chunk is not found as a parameter. When the selected chunk is found (step S65; Yes), the duplication determination unit 31 returns to step S23 with the fact that the selected chunk is found as a parameter.

[Effect of Example 2]
According to the second embodiment, the communication device 1 includes the memory 11 and the disk 14. The communication device 1 stores the data to be processed in one of the caches of the memory 11 and the disk 14 in accordance with the usage ratio of the memory 11. According to such a configuration, the communication device 1 uses the disk 14 to store data even when the capacity of the memory 11 is small. Therefore, the communication device 1 can realize deduplication without losing a part of data whose data duplication characteristic is RLC (rare and long chunk series).

[Others]
Note that the communication device 1 is equipped with functions such as the above-described duplication determination unit 23, new registration unit 24, duplication management unit 25, and cache migration unit 26 in an information processing apparatus such as a known personal computer or workstation. Can be realized.

  Moreover, in the said Example 1, 2, the communication apparatus 1 hierarchizes the cache preservation | save area | region which memorize | stores the data transmitted in the past. When the communication device 1 moves the data to a predetermined lower cache (for example, the disk 12 or the SSD 13) according to the data duplication characteristic at the timing when the upper cache (for example, the memory 11) is full. Explained. However, the present invention is not limited to the case of the communication device 1, and the information processing device hierarchizes a cache storage area for storing data processed in the past. When the information processing apparatus moves the data to a predetermined lower cache (for example, the disk 12 or the SSD 13) according to the data duplication characteristic at the timing when the upper cache (for example, the memory 11) is full. It may be. For example, the present invention may be applied to a cache memory between the CPU and the main storage device in the information processing apparatus. This cache memory may be used as a cache storage area.

  In addition, each component of the illustrated apparatus does not necessarily need to be physically configured as illustrated. In other words, the specific mode of device distribution / integration is not limited to that shown in the figure, and all or part of the device is functionally or physically distributed / integrated in an arbitrary unit according to various loads or usage conditions. Can be configured. For example, the chunk division unit 21 and the hash calculation unit 22 may be integrated as one unit. On the other hand, the cache transfer unit 26 may be distributed to a determination unit that determines whether or not the cache is full and a transfer unit that moves the corresponding data when the cache is full. In addition, the storage unit 10 may be stored in an external device of the communication device 1, or the external device storing the storage unit 10 may be connected to the communication device 1 via a network.

  The various processes described in the above embodiments can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a cache storage program that realizes the same function as that of the communication device 1 illustrated in FIG. 1 will be described. FIG. 10 is a diagram illustrating an example of a computer that executes a cache storage program.

  As illustrated in FIG. 10, the computer 200 includes a CPU 203 that executes various arithmetic processes, an input device 215 that receives input of data from the user, and a display control unit 207 that controls the display device 209. The computer 200 also includes a drive device 213 that reads a program and the like from a storage medium, and a communication control unit 217 that exchanges data with other computers via a network. The computer 200 also includes a memory 201 that temporarily stores various types of information and an HDD 205. The memory 201, CPU 203, HDD 205, display control unit 207, drive device 213, input device 215, and communication control unit 217 are connected by a bus 219.

  The drive device 213 is a device for a removable disk 141, for example. The HDD 205 stores a cache storage program 205a and cache storage related information 205b.

  The CPU 203 reads the cache storage program 205a, expands it in the memory 201, and executes it as a process. Such a process corresponds to each functional unit of the communication device 1. The cache storage related information 205 b corresponds to the duplication management information 111. For example, the removable disk 141 stores each piece of information such as the duplication management information 111.

  Note that the cache storage program 205a is not necessarily stored in the HDD 205 from the beginning. For example, the program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into the computer 200. Then, the computer 200 may read and execute the cache storage program 205a from these.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary note 1)
If the data to be processed is new, store the data to be processed in the first cache memory;
For each data stored in the first cache memory, the number of times of duplication that overlaps the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as a duplication history And
When processing new data, if the number of data stored in the first cache memory is the upper limit, the predetermined data to be deleted is extracted,
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A cache storage program characterized by causing processing to be executed.

(Additional remark 2) When the continuous duplication length of the data to be deleted is larger than a first threshold, the process to move the data to be deleted at a lower speed among the second cache memory and the third cache memory. The cache storage program according to appendix 1, wherein a process of moving to a second cache memory is executed.

(Supplementary Note 3) The moving process is performed when the continuous duplication length of the data to be deleted is equal to or less than a first threshold and the duplication count of the data to be deleted is larger than a second threshold. The cache storage program according to appendix 1, wherein a process of moving target data to a high-speed third cache memory among the second cache memory and the third cache memory is executed.

(Supplementary Note 4) The process of moving, when the continuous duplication length of the data to be deleted is less than or equal to a first threshold, and when the number of duplications of the data to be deleted is less than or equal to a second threshold, The cache storage program according to appendix 1, wherein processing for discarding data to be deleted is executed.

(Supplementary Note 5) When the first cache memory has a primary cache memory and a secondary cache memory, when the processing target data is stored in the first cache memory, use of the primary cache memory The cache storage program according to appendix 1, wherein a process of storing the data to be processed in either the primary cache memory or the secondary cache memory is executed according to a ratio.

(Appendix 6)
If the data to be processed is new, store the data to be processed in the first cache memory;
For each data stored in the first cache memory, the number of times of duplication that overlaps the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as a duplication history And
When processing new data, if the number of data stored in the first cache memory is the upper limit, the predetermined data to be deleted is extracted,
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A cache storage method characterized by executing each processing.

(Supplementary Note 7) When the data to be processed is new, a registration unit that registers the data to be processed in the first cache memory;
For each data stored in the first cache memory, the number of times of duplication that overlaps the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as a duplication history A management department to
When processing new data, when the number of data stored in the first cache memory is an upper limit, an extraction unit that extracts predetermined deletion target data;
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A moving part;
An information processing apparatus comprising:

(Supplementary Note 8) When the data to be transmitted is new, a registration unit that registers the data to be transmitted in the first cache memory;
For each data stored in the first cache memory, the number of times of duplication overlapping with the data to be transmitted and the continuous duplication length indicating the length of duplication when overlapping with the data to be transmitted are managed as a duplication history. A management department to
When processing new data, when the number of data stored in the first cache memory is an upper limit, an extraction unit that extracts predetermined deletion target data;
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A moving part;
A communication apparatus comprising:

DESCRIPTION OF SYMBOLS 1 Communication apparatus 10 Memory | storage part 11 Memory 111,211 Duplication management information 12 Disk 13 SSD
DESCRIPTION OF SYMBOLS 20 Control part 21 Chunk division part 22 Hash calculation part 23,31 Duplication determination part 24 New registration part 25 Duplication management part 26 Cache movement part 27 Transmission part 32 Sampling chunk registration part

Claims (7)

On the computer,
If the data to be processed is new, store the data to be processed in the first cache memory;
For each data stored in the first cache memory, the number of times of duplication that overlaps the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as a duplication history And
When processing new data, if the number of data stored in the first cache memory is the upper limit, the predetermined data to be deleted is extracted,
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A cache storage program characterized by causing processing to be executed. In the moving process, when the continuous duplication length of the data to be deleted is larger than a first threshold value, the data to be deleted is transferred to a second cache having a low speed among the second cache memory and the third cache memory. The cache storage program according to claim 1, wherein a process of moving to a memory is executed. The moving process is performed when the continuous duplication length of the data to be deleted is equal to or smaller than a first threshold and the duplication frequency of the data to be deleted is larger than a second threshold. The cache storage program according to claim 1, wherein a process of moving to a high-speed third cache memory out of the second cache memory and the third cache memory is executed. The moving process includes the deletion target data when the continuous duplication length of the deletion target data is equal to or smaller than a first threshold and when the number of times of duplication of the deletion target data is equal to or smaller than a second threshold. The cache storage program according to claim 1, wherein a process for discarding is executed. When the first cache memory includes a primary cache memory and a secondary cache memory, when storing the data to be processed in the first cache memory, the first cache memory depends on a usage ratio of the primary cache memory. The cache storage program according to claim 1, wherein a process of storing the processing target data in any one of the primary cache memory and the secondary cache memory is executed. Computer
If the data to be processed is new, store the data to be processed in the first cache memory;
For each data stored in the first cache memory, the number of times of duplication that overlaps the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as a duplication history And
When processing new data, if the number of data stored in the first cache memory is the upper limit, the predetermined data to be deleted is extracted,
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A cache storage method characterized by executing each processing. A registration unit for registering the processing target data in the first cache memory when the processing target data is new;
For each data stored in the first cache memory, the number of times of duplication that overlaps the data to be processed and the continuous duplication length that indicates the length of duplication that continues when overlapping with the data to be processed are managed as a duplication history A management department to
When processing new data, when the number of data stored in the first cache memory is an upper limit, an extraction unit that extracts predetermined deletion target data;
The data to be deleted is moved to one of the second cache memory and the third cache memory according to the number of times of duplication and the continuous duplication length of the data to be deleted managed in the duplication history. A moving part;
An information processing apparatus comprising:

Images