JP2013088920A - Computer system and data management method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform management so that a plurality of applications share the same value in key value type data arranged in a distributed memory cache.SOLUTION: The computer system includes: a plurality of first computers for storing data; and a second computer for managing the data stored in each of the plurality of first computers. The computer system includes a storage generated by integrating storage areas provided by the plurality of first computers. The storage stores first divided data including first access information and a first retrieval key and a value of a first file. The second computer generates a second retrieval key when receiving an arrangement request of divided data of the first file from the application, determines whether or not the value of the first file is shared value, and stores second divided data including the second retrieval key and the first access information in the storage when it is determined that the value of the first file is the shared value.

Description

  The present invention relates to a computer system and a data management method including a storage for distributing and arranging data.

  In recent years, the amount of data to be processed by application programs in computer systems has increased explosively. Due to an increase in processing time accompanying an increase in the amount of data handled by a computer system, a problem has arisen that processing such as batch jobs does not end within a predetermined time. For this reason, it is required to process a large amount of data at high speed, and in order to realize this, for example, a request for speeding up the processing by causing a large number of servers to process a large amount of data in parallel has increased. ing.

  Conventional application programs generally handle data in a file format or a database (DB) table format. Each application program uses different files and tables. In particular, in a basic business process in which a mainframe is used, an application program is created using COBOL as a programming language. In the application program as described above, data is handled with a file and a table as a set of records.

  A record is a basic unit of data processed by an application program, and the application program inputs and outputs data in record units. One record stores a series of related information, and each item of the information is called a field. Taking information handled by a financial institution as an example, one item of transaction information corresponds to a record, and each item such as an account number, a store number, and a product code corresponds to a field.

  The application program sequentially reads records one by one from the file and executes predetermined processing. When a conventional application program processes data in the format as described above in parallel, a method of executing processing by dividing the data into records can be considered.

  The reason for dividing the data is that the application program reads the records one by one and executes the process. Therefore, simply creating a copy of the data and processing the data by multiple servers, the processing amount of each server is This is because it does not change.

  From the viewpoint of dividing data in units of records, there is a distributed database technique that divides a database in units of records using keys (see, for example, Patent Document 1). Japanese Patent Application Laid-Open No. 2005-228561 describes a technique for realizing parallel processing by dividing data stored in a database for each record by a key range.

  Further, as a technique related to parallel processing of data, there is a technique for leveling a load in a non-shared database system that operates on a disk shared by each server (see, for example, Patent Document 2). Patent Document 2 is composed of a plurality of database servers, allows data to be transferred between database servers by subdividing the data area and changing the allocation of each data area to the database server. Techniques for improving the performance are described.

  On the other hand, as a technique for processing a large amount of data at high speed using a large number of servers, for example, a distributed memory cache technique as shown in Non-Patent Document 1 has been proposed. The distributed memory cache technology is a technology for composing a memory space for storing a large amount of data by integrating memories of a plurality of servers. The object is to realize parallel processing by dividing and arranging data and speeding up input / output by holding data in a memory.

  In the distributed memory technology, a key / value type data format is adopted to distribute a large amount of data to a plurality of servers. The key / value type data has a data structure in which a key that is an identifier of data and a value (value) that is a data body are associated with each other, and is managed by a combination of [key and value]. One key / value type data is also described as an entry.

  In the distributed memory technology, data is divided and arranged in a plurality of servers according to a key range (key range), and the data arranged in a divided manner is processed in parallel by an application running on each server, thereby speeding up the processing. .

JP-A-5-334165 JP 2007-25735 A

“GemFire Enterprise” Technical White Paper 2007 GemStone Systems Inc.

  When the data source is data having a structure such as a file composed of a table format or a record, a field to be a key is different for each application program. For example, an application program focused on a store number handles key / value type data using the store number as a key, whereas an application program focused on an account number handles key / value type data using the account number as a key.

  Therefore, even in the same record, the key / value type data is different for each application. At this time, when the value is updated from a certain application program, the update may not be reflected in the value of the key / value data handled by another application. To reflect the value update, reflect the value update to the data source corresponding to the updated entry, and specify another key field for the application program that references the record. , Need to regenerate the entry.

  When two application programs update fields of different records of the same record, the fields updated by one application program may be overwritten when writing back to the data source.

  A typical example of the present invention is as follows. A plurality of first computers having a first processor, a first memory connected to the first processor, and a first network interface connected to the first processor, and storing data; , A second processor, a second memory connected to the second processor, and a second network interface connected to the second processor, stored in each of the plurality of first computers. A second computer that manages the data to be stored, the computer system comprising a storage generated by integrating storage areas provided by the plurality of first computers, and the second computer system. The computer divides a file including a plurality of records, and stores the divided data in which the search key is associated with the value indicating the content of the record. A loader processing unit that stores the information in a distributed manner and a file storage unit that stores the file, and storage configuration information that includes information on the storage area that configures the storage, and is stored in the storage Corresponding information for managing the correspondence between the divided data and the file is stored, and each of the plurality of first computers includes an application that processes data in units of file data, and a storage management unit that manages the storage And stores divided data management information for managing the divided data, and the storage includes first access information for accessing the value of the first file and a first search key. The divided data and the value of the first file are stored, and the second computer receives the second data from the application. The second search key designated by the application is generated when a request for placement of the divided data of the first file including search key information is received, and the value of the first file includes a plurality of values. It is determined whether or not it is a shared value shared by an application, and if it is determined that the value of the first file is a shared value, the second search key and the first access information are included. The second divided data is stored in the storage.

  According to the present invention, it is not necessary to regenerate an entry from a file by configuring the entries so that the same value can be shared among the entries generated by a plurality of applications. Further, since the same value can be shared, the update of one entry can be reflected in other entries referred to by other applications.

It is a block diagram which shows the structural example of the computer system in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the data source in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the distributed memory cache structure information in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the data source information in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the correspondence information in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the record definition information in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the source program of UAP in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the data structure of the entry in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the shared value management information in the 1st Embodiment of this invention. It is explanatory drawing which shows the structural example of the shared area management information in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the logical data structure of the entry in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the logical data structure of the entry in the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the logical data structure of the entry in the 1st Embodiment of this invention. It is a flowchart explaining the detail of the load process in the 1st Embodiment of this invention. It is a flowchart explaining the detail of the unload process in the 1st Embodiment of this invention. It is a flowchart explaining the detail of the unload process in the 1st Embodiment of this invention. It is a block diagram which shows the structural example of the computer system in the 2nd Embodiment of this invention. The present invention is a block diagram showing a configuration example of a computer system according to a third embodiment. It is explanatory drawing which shows the structural example of the entry in the 3rd Embodiment of this invention. It is explanatory drawing which shows the structural example of the access statistical information in the 3rd Embodiment of this invention. It is explanatory drawing which shows the structural example of the transition condition information in the 3rd Embodiment of this invention. It is a flowchart explaining the data migration process in the 3rd Embodiment of this invention. It is explanatory drawing which shows an example of the logical data structure of the entry in the 3rd Embodiment of this invention. It is explanatory drawing which shows an example of the logical data structure of the entry in the 3rd Embodiment of this invention.

[First Embodiment]
FIG. 1 is a block diagram illustrating a configuration example of a computer system according to the first embodiment of this invention.

  The computer system according to the first embodiment includes a host computer 101, a plurality of host computers 102, a storage device 103, and a shared storage device 105.

  The host computer 101 is connected to the host computer 102 via the network 104. Each host computer 102 is connected to the shared storage device 105. The network 104 may be a LAN, a WAN, or the like, but the present invention is not limited to the type of the network 104. Note that the shared storage device 105 may be connected via the network 104.

  In the present embodiment, a distributed memory cache is generated by integrating storage areas included in a computer system. The distributed memory cache according to this embodiment includes a distributed area 171 generated from a memory area included in each host computer 102 and a shared area 172 generated from a storage area of the shared storage device 105.

  Key / value type data (entry 191) is stored in the distributed memory cache. In the present embodiment, an entry 191 is stored in the distribution area 171. The entry 191 includes at least a key and does not necessarily include a value. In this case, the entry 191 includes information for accessing the value 192. In the present embodiment, the value 192 is stored in either the distributed area 171 or the shared area 172. Details of the entry 191 will be described later.

  The distributed memory cache can be accessed from the host computer 102 in the same manner as a shared device. By storing the data to be processed in the distributed memory cache, the input / output processing becomes faster than when the data is stored in the storage apparatus 103.

  The host computer 101 is connected to the storage apparatus 103. The storage device 103 stores a data source 117 to be processed. The storage apparatus 103 may be anything as long as it can hold the data source 117 permanently. The storage device 103 may be, for example, a storage system including a plurality of storage media such as HDDs, a semiconductor disk device using a flash memory as a storage medium, an optical disk device, and the like.

  In this embodiment, the data source 117 to be processed is described as using files and DB tables. However, the present invention is not limited to the storage format of the data source 117, and the data storage format is not particularly limited as long as the data can be held permanently such as a key value store (KVS).

  The host computer 101 includes a processor 111, a memory 113, and interfaces (I / F) 115A and 115B. The host computer 101 is connected to the storage apparatus 103 via the interface 115A, and the host computer 101 is connected to the host computer 102 via the interface 115B.

  The processor 111 executes a program stored in the memory 113. When the processor 111 executes a program stored in the memory 113, a function of the host computer 101 described later is realized.

  The memory 113 stores a program executed by the processor 111 and data necessary for executing the program. The memory 113 may be a semiconductor memory such as a DRAM, and can be accessed at a higher speed than the storage device 103.

  The memory 113 according to the present embodiment stores programs for realizing the data source management unit 121 and the loader unit 131, and also stores distributed memory cache configuration information 181, data source information 182, and correspondence information 183.

  The data source management unit 121 manages the data source 117 stored in the storage apparatus 103 and executes input / output processing for the data source 117.

  The loader unit 131 cooperates with the data source management unit 121 to distribute and arrange the data sources 117 stored in the storage apparatus 103 in the distributed memory cache, and also to store the key / value type data (entry) stored in the distributed memory cache. 191) is stored in the storage apparatus 103. The loader unit 131 executes the processing described above with reference to the distributed memory cache configuration information 181, the data source information 182, and the correspondence information 183.

  Hereinafter, the process of distributing and arranging the data sources 117 stored in the storage apparatus 103 in the distributed memory cache is referred to as a load process. In addition, a process for storing key / value data stored in the distributed memory cache in the storage apparatus 103 is referred to as an unload process.

  The distributed memory cache configuration information 181 stores information related to the distributed area 171 and the shared area 172 constituting the distributed memory cache. Details of the distributed memory cache configuration information 181 will be described later with reference to FIG.

  The data source information 182 stores information about the data source 117 such as a file and a DB table. Details of the data source information 182 will be described later with reference to FIG.

  The correspondence information 183 stores information on the correspondence relationship between the data source 117 and the entry 191 arranged in the distributed memory cache. Details of the correspondence information 183 will be described later with reference to FIG.

  The program and data stored in the memory 113 need not always be stored in the memory 113, and may be stored in the storage device 103 or an external storage device (not shown). In this case, the program or data is read from the storage device 103 or the external storage device to the memory 113 as necessary. In addition, when reading data, a part or all of data can be read.

  The data source management unit 121 and the loader unit 131 are realized by programs, but the present invention is not limited to this. For example, the functions of the data source management unit 121 and the loader unit 131 may be realized using dedicated hardware.

  The host computer 102 includes a processor 112, a memory 114, and an interface (I / F) 116. The host computer 102 is connected to the host computer 101 and other host computers 102 via an interface 116A. The host computer 102 is connected to the shared storage device 105 via the interface 116B.

  Here, the host computer 102 is described as having the same configuration. However, the host computer 102 is not necessarily required to have the same configuration as long as the functions and processes described below can be realized.

  The processor 112 executes a program stored in the memory 114. The processor 112 executes a program stored in the memory 114, thereby realizing a function of the host computer 102 described later.

  The memory 114 stores a program executed by the processor 112 and data necessary for executing the program. The memory 114 may be a semiconductor memory such as a DRAM, and can be accessed at a higher speed than the storage device 103.

  The memory 114 of this embodiment stores a program for realizing the distributed memory cache management unit 141 and an application program (UAP) 161. The memory 114 also includes a distributed area 171 that constitutes a distributed memory cache.

  The distributed memory cache management unit 141 manages the distributed area 171 and the shared area 172 constituting the distributed memory cache in cooperation with the distributed memory cache management unit 141 of another host computer 102. The distributed memory cache management unit 141 controls access to the distributed memory cache.

  The distributed memory cache management unit 141 includes an entry arrangement management unit 142, a shared area access management unit 143, shared value management information 144, and shared area management information 145. Although not shown, the distributed memory cache management unit 141 includes file management information that is information for managing the location of the entry 191.

  The distributed memory cache management unit (not shown) provides a distributed memory cache management function. The entry arrangement management unit 142 manages a storage area in which the entry 191 is stored. The shared area access management unit 143 and the access to the shared area 172 are managed.

  The shared value management information 144 stores information regarding the value 192 stored in the shared area 172. Details of the shared value management information 144 will be described later with reference to FIG. The shared area management information 145 stores management information for the shared area 172. Details of the shared area management information 145 will be described later with reference to FIG. Further, the file management information (not shown) stores data source identifiers, information indicating entry storage locations, and the like.

  Note that the program and data stored in the memory 114 need not always be stored in the memory 114, and may be stored in a storage device (not shown) or an external storage device (not shown). In this case, a program or data is read from the disk device or the external storage device to the memory 114 as necessary. In addition, when reading data, a part or all of data can be read.

  The data source management unit 121 and the distributed memory cache management unit 141 described above may be provided as a part of an operating system (OS) (not shown) or an input / output library used by a user application program (not shown). Further, the functions provided in the data source management unit 121 and the distributed memory cache management unit 141 may be realized using dedicated hardware.

  The shared storage device 105 uses a device having better random access characteristics than a normal disk device, such as a semiconductor disk device using a flash memory as a storage medium, but is a storage device that can be physically or logically shared. Any type of storage medium can be used. In this embodiment, a shared area 172 constituting a distributed memory cache is secured in the storage area of the shared storage device 105.

  Note that the host computer 102 in FIG. 1 does not have to be a physical computer, and may be a logical computer. In this case, computer resources such as the processor 112, the memory 114, and the interface 116 are allocated to the logical computer as logical computer resources by a virtualization program (not shown).

  FIG. 2 is an explanatory diagram illustrating a configuration example of the data source 117 according to the first embodiment of this invention. In FIG. 2, a case where the data source 117 is a file will be described.

  The file 2100 is composed of a plurality of records as basic units of data processed by the UAP 161. In the example illustrated in FIG. 2, the file 2100 includes a record 2101, a record 2102, a record 2103, and a record 2104.

  Each record 2101, 2102, 2103, 2104 is composed of a plurality of fields 2111, 2112, 2113 for storing a series of related information. In the example illustrated in FIG. 2, each record includes a field 2111, a field 2112, and a field 2113.

  In general, within the constraints of the system, a file can contain any number of records, and a record can contain any number of fields. For example, in the case of data used in merchandise transaction business, a record is composed of transaction information in one transaction, and individual information (data) such as account number, store number, and merchandise code is recorded in the field. .

  The UAP 161 executes I / O processing with a record as a processing unit, and executes processing by inputting / outputting records one by one to / from a file. In this embodiment, the file 2100 is divided into record units using an arbitrary field as a key. The record is divided so as to be adapted to the processing executed by the UAP 161.

  The divided data of the file 2100, that is, key / value type data is assigned to the UAP 161 on the plurality of host computers 102, and each UAP 161 executes processing using the assigned key / value type data. As described above, when a plurality of UAPs 161 process key / value data obtained by dividing one file 2100 in parallel, the amount of data processed by one UAP 161 can be reduced. Therefore, the processing speed can be increased.

  FIG. 3 is an explanatory diagram illustrating a configuration example of the distributed memory cache configuration information 181 according to the first embodiment of this invention.

  The distributed memory cache configuration information 181 includes a total size 201, a distributed area size 202, a distributed area usage rate 203, a shared area size 204, and a shared area usage rate 205.

  The total size 201 stores the size of the entire distributed memory cache. Specifically, the sum of the size of the distributed area 171 and the size of the shared area 172 is stored. In the present embodiment, the value obtained by adding the distributed area size 202 and the shared area size 204 is the size of the distributed memory cache.

  The distributed area size 202 stores the total size of all the distributed areas 171 constituting the distributed memory cache. Specifically, the distribution area size 202 stores the sum of the sizes of the distribution areas 171 of the host computers 102. The distributed area usage rate 203 stores the usage rate of the distributed area 171.

  The shared area size 204 stores the size of the shared area 172. The shared area usage rate 205 stores the usage rate of the shared area 172.

  The distributed memory cache configuration information 181 is used to determine whether or not there is a sufficient storage area when executing the load process. Although not described in detail, when there is not enough storage area, processing for securing the storage area is executed. As processing for securing the storage area, for example, unload processing, migration processing of the entry 191 between the distributed area 171 and the shared area 172, and the like can be considered.

  FIG. 4 is an explanatory diagram illustrating a configuration example of the data source information 182 according to the first embodiment of this invention.

  The data source information 182 includes data source identification information 301, record definition information 302, and arrangement designation information 303.

  The data source identification information 301 stores identification information for uniquely identifying a plurality of data sources 117 stored in the storage apparatus 103. The data source identification information 301 may store any information as long as the data source 117 can be uniquely identified. For example, the data source identification information 301 stores a full path file name or a table name in the DB.

  Depending on the management method of the data source 117, identification information of the storage apparatus 103 and the host computer 101 that hold the data source 117 may be required. In such a case, the data source identification information 301 includes identification information of the storage apparatus 103 and the host computer 101.

  Note that the data source information 182 corresponding to the data source 117 only needs to be uniquely identified. Therefore, when the data source attribute or the like is managed so as to be associated with the data source on a one-to-one basis, the data source identification information 301 is not necessary.

  The record definition information 302 stores information related to the record structure in the data source 117. Details of the record definition information 302 will be described later with reference to FIG.

  The arrangement designation information 303 stores information for designating an arrangement destination storage area when executing the load process. The placement designation information 303 stores information such as “distributed area”, “shared area”, or “no designation”, for example.

  When “distributed area” is stored, according to the key of the entry 191, the entry 191 and the value 192 are preferentially arranged in the distributed area 171 of the host computer 102 that manages the entry 191 of a predetermined key range. When the “shared area” is stored, the entry 191 is arranged in the distributed area 171 of the predetermined host computer 102, and the value 192 is preferentially arranged in the shared area 172. Further, when “not specified” is stored, it indicates that the storage area of the storage destination is not specified.

  FIG. 5 is an explanatory diagram illustrating a configuration example of the correspondence information 183 according to the first embodiment of this invention.

  The correspondence information 183 includes data source identification information 401, reference number 402, area usage information 403, and load identification information 404.

  The data source identification information 401 stores identification information for uniquely identifying the data source 117. The data source identification information 401 is the same as the data source identification information 301.

  The reference number 402 stores the number of times the data source 117 has been loaded into the distributed memory cache. That is, the reference number 402 indicates the number of key / value type data handled by the UAP 161. In the present embodiment, it is determined whether one data source is shared by a plurality of UAPs 161 based on the reference number 402.

  The area usage information 403 stores the type of storage area in which the value 192 is stored. That is, the area usage information 403 stores information indicating whether the value 192 is stored in the distributed area 171 or the shared area 172.

  The load identification information 404 stores identification information returned by the loader unit 131 each time a load process is executed. In this embodiment, when executing the unload process, it is determined which entry 191 among the entries 191 stored in the distributed memory cache is the target of the unload process based on the load identification information 405. .

  FIG. 6 is an explanatory diagram illustrating a configuration example of the record definition information 302 according to the first embodiment of this invention.

  The record definition information 302 stores information regarding the structure of records in the data source 117. In the present embodiment, the data source 117 can be divided into record units based on the record definition information 302. The record definition information 302 includes a record structure 501 and a field structure 502.

  The record structure 501 stores information for grasping the record structure in the data source 117, and includes a record delimiter 511, a record type 512, and a record length 513.

  The record delimiter 511 stores information indicating a character code that separates the record from other records. For the record delimiter 511, for example, a character code representing a line feed may be used.

  The record type 512 stores information indicating whether the record in the data source 117 is a fixed-length record or a variable-length record. When information indicating a fixed-length record is stored in the record type 512, the data source 117 is composed of records having the same and predetermined length. When information indicating a variable-length record is stored in the record type 512, the data source 117 is composed of records having different record lengths.

  The record length 513 stores information indicating the length of one record when information indicating a fixed-length record is stored in the record type 512.

  Note that the record configuration 501 only needs to include information capable of grasping the structure of the record, and does not need to include all information of the record delimiter 511, the record type 512, and the record length 513. For example, in the case of a fixed-length record, the record delimiter 511 may not be included in the record configuration 501.

  The field configuration 502 stores information for identifying a field included in the record, and includes a field delimiter 521, a field number 522, and field information 523.

  The field delimiter 521 stores information indicating a character code that separates the field from other fields. For the field delimiter 521, for example, a character code representing a blank can be used.

  The field number 522 stores the number of fields included in one record.

  The field information 523 stores information related to data recorded in the corresponding field, and includes a field type 531, a field length 532, and a description format 533.

  The field type 531 stores information indicating whether a corresponding field is a variable length field or a fixed length field when information indicating a variable length record is stored in the record type 512.

  The field length 532 stores information indicating the size of the corresponding field. The description format 533 stores a description format of data recorded in a corresponding field such as ASCII or binary.

  Since the field configuration 502 only needs to be able to grasp the fields included in the record, it is not necessary to include all information of the field delimiter 521, the field number 522, and the field information 523. For example, if the field length 532 of the field information 523 is specified, the information of the field delimiter 521 may not be included in the field configuration 502.

  When the data source 117 is composed of fixed-length records, each record can be recognized by the value set in the record length 513. On the other hand, when the data source 117 is composed of variable-length records, a field for recording the size of the record is provided at the top of each record, and the delimiter of the record can be determined based on the field. . If the record is a variable length record, the first field is identified based on the information stored in the field configuration 502, and the record size can be calculated. After the record is recognized, the field can be grasped by referring to the field number 522 of the field configuration 502 and the field length 532 of the field information 523.

  FIG. 7 is an explanatory diagram illustrating an example of the source program of the UAP 161 according to the first embodiment of this invention.

  FIG. 7 shows the source code of the UAP 161 described using the COBOL language. In UAP 161 described using the COBOL language, a record structure of a file as a data source is defined in a program.

  In the example shown in FIG. 7, the file structure is defined in FILE SECTION 602 of DATA DIVISION. Each file used in the program is defined by one file description entry (FD) followed by one or more record description entries. In the present embodiment, information described in the FILE SECTION 602 is stored in the record structure 501 and the field structure 502 of the record definition information 302.

  FIG. 8 is an explanatory diagram illustrating an example of a data configuration of the entry 191 according to the first embodiment of this invention.

  The entry 191 stores information for the distributed memory cache management unit 141 to acquire the value 192 corresponding to the key specified by the UAP 161. The entry 191 includes a key 701, area usage information 702, storage location information 703, and shared value management information pointer 705.

  The key 701 stores the key in the entry 191.

  The area usage information 702 stores information related to the storage area in which the value 192 corresponding to the entry 191 is arranged. In this embodiment, the area usage information 702 stores either “distributed area” or “shared area”. The area usage information 702 is set when the loader unit 131 arranges the value 192 or when the distributed memory cache management unit 141 changes the arrangement location of the value 192.

  When “distributed area” is stored in the area usage information 702, it indicates that the value 192 is stored in the distributed area 171. That is, it indicates that there is no UAP 161 that handles the same data source 117. On the other hand, when “shared area” is stored in the area usage information 702, it indicates that the value 192 is stored in the shared area 172. That is, it indicates that a plurality of UAPs 161 handle the same data source 117.

  The logical data structure of the entry 191 will be described later with reference to FIGS.

  The storage location information 703 stores information for accessing the value, that is, information indicating the storage location of the value.

  When “distributed area” is stored in the area usage information 702, the storage location information 703 stores the value 192 stored in the distributed area 171 of the host computer 102 or the value 192 stored in the distributed area 171. Information indicating the position is stored. As the information indicating the storage position of the value 192, a memory address or the like can be considered.

  When “shared area” is stored in the area use information 702, information indicating the storage position of the value 192 stored in the shared area 172 is stored in the storage position information 703.

  The shared value management information pointer 704 stores a pointer to the shared value management information 144 that manages the value 192 stored in the shared area 172.

  Hereinafter, the value 192 stored in the shared area is also referred to as a shared value 192.

  FIG. 9 is an explanatory diagram illustrating a configuration example of the shared value management information 144 according to the first embodiment of this invention.

  The shared value management information 144 is generated when the entry 191 is generated in the load process. The shared value management information 144 is deleted when the entry 191 is deleted from the distributed memory cache.

  The shared value management information 144 includes data source identification information 801, record identification information 802, storage location information 803, the number of shares 804, an entry pointer 805, and a shared area management information pointer 806.

  The data source identification information 801 and the record identification information 802 are information for associating the entry 191 with the record of the data source 117. Specifically, the data source identification information 801 stores identification information for uniquely identifying a plurality of data sources 117 stored in the storage apparatus 103. The data source identification information 801 is the same information as the data source identification information 301.

  The record identification information 802 stores information for identifying the record of the data source 117. As the record identification information 802, for example, the record number or line number of the record included in the data source 117 can be used, but any information may be stored as long as the record can be identified.

  The storage location information 803 stores information indicating the storage location of the value 192 of the entry 191. The storage location information 803 is the same as the storage location information 703.

  The sharing number 804 stores the number of UAPs 161 that share the value 192.

  The entry pointer 805 stores a pointer indicating the storage position of the entry 191.

  The shared area management information pointer 806 stores a pointer to the shared area management information 145 that is information for controlling the consistency of the value 192 stored in the shared area 172.

  FIG. 10 is an explanatory diagram illustrating a configuration example of the shared area management information 145 according to the first embodiment of this invention.

  The shared area management information 145 stores information necessary for maintaining the consistency of the value 192 stored in the shared area 172. The shared area management information 145 includes storage location information 901, shared value management information pointer 902, update right management information 903, and reference right management information 904.

  The storage location information 901 stores information indicating the storage location of the shared value 192. Specifically, information indicating the storage position of the value 192 stored in the shared area 172 is stored.

  The shared value management information pointer 902 stores a pointer to the shared value management information 144. The shared value management information pointer 902 is the same as the shared value management information pointer 704.

  The update right management information 903 stores identification information of the host computer 102 having the authority to update the shared value 192. The reference right management information 904 stores identification information of the host computer 102 that has the authority to reference the shared value 192. In the update right management information 903 and the reference right management information 904, for example, the IP address and the Mac address of the host computer 102 are stored. As long as the host computer 102 can be identified, any information may be stored in the update right management information 903 and the reference right management information 904.

  In the present embodiment, the shared area management information 145 is pointed from the shared value management information 144, but any data structure that can maintain the consistency of the shared value 192 may be used. . In this case, the distributed memory cache management unit 141 does not need to manage. For example, there is a method in which a shared file system management program controls the consistency of the shared value 192 by associating the shared value 192 with a file of the shared file system using a shared file system constructed in the shared storage device 105. Conceivable.

  Hereinafter, the structure of the entry 191 in the present invention will be described.

  FIGS. 11, 12, and 13 are explanatory diagrams illustrating an example of a logical data configuration of the entry 191 according to the first embodiment of this invention.

  FIG. 11 shows a logical data structure of the entry 191 when the value 192 is stored in the distribution area 171. In the example shown in FIG. 11, the entry 191 is recognized as data having a key 701, area usage information 702, and a value 192 as one set.

  When the distributed memory cache management unit 141 receives an access request including the designated key 701 from the UAP 161, the value 192 is obtained by referring to the area usage information 702 of the entry 191 corresponding to the designated key. 171 is determined to be stored. Further, the distributed memory cache management unit 141 accesses the predetermined value 192 and responds to the UAP 161.

  FIG. 12 shows a logical data structure of the entry 191 when the value 192 is stored in the shared area 172. In the example shown in FIG. 12, the entry 191 is recognized as data with the key 701, the area usage information 702, the shared value storage location information 1103, and the shared value 192 as one set.

  When the distributed memory cache management unit 141 receives an access including the specified key 701 from the UAP 161, the value 192 is changed to the shared area 172 by referring to the area usage information 702 of the entry 191 corresponding to the specified key. It is determined that it is stored in. Further, the distributed memory cache management unit 141 refers to the shared value storage location information 1103, identifies the storage location of the shared value 192, accesses the shared value 192, and responds to the UAP 161. As the shared value storage location information 1103, for example, a block address indicating the storage location of data in the shared storage device 105 and size information of value may be considered. However, the shared value storage location information 1103 may be any information as long as the storage location of the shared value 192 can be uniquely identified.

  FIG. 13 shows a logical data structure of the entry 191 when stored in the shared area 172. The entry shown in FIG. 13 is different in that shared value identification information 1203 that uniquely identifies a shared value is used instead of the shared value storage location information 1103.

  The shared value identification information 1203 is, for example, a combination of the file name or data source identification information 301 in the shared file system constructed on the shared storage device 105 and the record number (or line number) in the data source 117. Conceivable.

  As shown in FIG. 13, the shared value 192 is also stored in the distributed memory cache so as to become the key / value type data entry 191, whereby the entry 191 having the same data structure is stored in the distributed area 171 and the shared area 172. Can be stored.

  FIG. 14 is a flowchart illustrating details of the load process according to the first embodiment of the present invention.

  The loader process is executed by the loader unit 131. When loading data from the data source 117 to the distributed memory cache, the loader unit 131 starts a loader process described below.

  When executing the load process, the data source 117 to be loaded and information to be used as a key are designated. The information that should be the key depends on what the UAP 161 focuses on. For example, in the case of the file 2100 shown in FIG.

  The loader unit 131 generates an entry based on the designated data source 117 to be loaded and information to be designated as the designated key (step S1401). Specifically, the following processing is executed.

  The loader unit 131 identifies the data source information 182 based on the identification information of the designated data source 117, and grasps the record structure of the data source 117 by referring to the record definition information 302 of the data source information 182.

  The loader unit 131 generates a key based on the grasped record structure and generates an entry 191. The loader unit 131 sets the generated key as the key 701 of the entry 191. The above is the process of step S1401.

  Next, the loader unit 131 updates the reference number 402 of the correspondence information 183 (step S1402). Specifically, the loader unit 131 increments the value of the reference number 402 by “1”. The loader unit 131 can determine whether the same data source 117 is currently loaded on the distributed memory cache based on the value of the reference number 402.

  The loader unit 131 determines whether an arrangement location of the designated data source 117 is designated (step S1403). Specifically, the following processing is executed.

  The loader unit 131 refers to the arrangement designation information 303 of the data source information 182 and determines whether or not an arrangement location is designated. When “distributed area” or “shared area” is stored in the arrangement designation information 303, it is determined that the arrangement location of the designated data source 117 is designated. On the other hand, when “no designation” is stored in the arrangement designation information 303, it is determined that the arrangement location of the designated data source 117 is not designated.

  When it is determined that the location of the specified data source 117 is not specified, the loader unit 131 determines whether the reference number 402 is “2” or more (step S1404). This is a process for the loader unit 131 to determine the arrangement location of the data source 117 when the arrangement location is not designated.

  When the reference number 402 is smaller than “2”, since the value 192 is not shared by the plurality of UAPs 161, the distribution area 171 is selected as the arrangement location. On the other hand, when the reference number 402 is “2” or more, since the value 192 is shared by the plurality of UAPs 161, the shared area 172 is selected as the arrangement location.

  If it is determined that the reference number 402 is “2” or more, the loader unit 131 determines whether or not the value 192 has been stored in the shared area 172 (step S1405). Specifically, the loader unit 131 refers to the area usage information 403 of the correspondence information 183 and determines whether or not the value 192 is stored in the shared area 172.

  When it is determined that the value 192 has already been stored in the shared area 172, the loader unit 131 places the entry 191 in the distributed area 171 (step S1407), and ends the process. Specifically, the following processing is executed.

  The loader unit 131 sets “shared area” in the area usage information 702 of the entry 191, and sets information for referring to the shared value 192 in the storage location information 703 of the entry 191. As the storage location information 703, it is conceivable to use the shared value storage location information 1103 or the shared value identification information 1203.

  Further, the loader unit 131 arranges the created entry 191 in the distribution area 171 of the predetermined host computer 102 according to the key range. In addition, the loader unit 131 generates shared value management information 144 and shared area management information 145 and transmits them to each host computer 102. The above is the process of step S1407.

  When it is determined that the value 192 has not been stored in the shared area 172, the loader unit 131 arranges the value 192 in the shared area 172, and further arranges the entry 191 in the distributed area 171 (step S1406). finish. Specifically, the following processing is executed.

  First, the loader unit 131 reserves an area for storing value in the shared area 172. The loader unit 131 reads data from the data source 117 and loads the shared value 192 into the shared area 172.

  Further, the loader unit 131 sets “shared area” in the area usage information 702 of the entry 191 and sets information for referring to the shared value 192 in the storage position information 703. Further, the loader unit 131 arranges the created entry 191 in the distribution area 171 of the predetermined host computer 102 according to the key range. In addition, the loader unit 131 generates shared value management information 144 and shared area management information 145 and transmits them to each host computer 102.

  If an entry 191 sharing the same value 192 already exists in the distributed area 171, the loader unit 131 copies the value 192 shared by the entry 191 from the distributed area 171 to the shared area 172. Thereafter, the loader unit 131 invalidates the area of the distributed area 171 in which the value 192 corresponding to the existing entry 191 is stored. Furthermore, “shared area” is set in the area usage information 702 of the new entry 191 and the existing entry 191, and information for referring to the shared value 192 is set in the storage location information 703. The new entry 191 is arranged in the distribution area 171 of the predetermined host computer 102 according to the key range.

  If it is determined in step S1403 that the arrangement location of the designated data source 117 is designated, the loader unit 131 determines whether or not the designated arrangement location is the shared area 172 (step S1408). . Specifically, the loader unit 131 determines whether the “shared area” is stored in the arrangement designation information 303 of the data source information 182.

  If it is determined that the designated placement location is the shared area 172, the loader unit 131 proceeds to step S1405.

  When it is determined that the designated arrangement location is not the shared area 172, the loader unit 131 arranges the entry 191 and the value 192 in the distribution area 171 (step S1409), and ends the process. At this time, the loader unit 131 sets “distributed area” in the area usage information 702 of the entry 191, and information for accessing the value 192 is set in the storage location information 703.

  When the distribution area 171 is designated as the arrangement location, the entry 191 is loaded into the distribution area 171 regardless of the number of data sources 117 loaded. As a result, like the conventional distributed memory cache, the entry 191 that does not share the value 192 among the plurality of UAPs 161 can be used.

  Note that the identification information that the loader unit 131 responds as a return value after the load process is completed is stored in the load identification information 405 of the correspondence information 183.

  In steps S1406 and S1407, the loader unit 131 may transmit necessary information to the host computer 102, and the distributed memory cache management unit 141 may generate the shared value management information 144 and the shared area management information 145.

  Since the access method to the entry 191 is the same as the conventional access method, description thereof is omitted.

  15A and 15B are flowcharts illustrating details of the unload processing according to the first embodiment of this invention.

  When data is unloaded from the distributed memory cache to the data source 117, the loader unit 131 executes unload processing described below. After the UAP 161 on the host computer 102 executes data processing, the loader unit 131 on the host computer 101 is executed when unloading from the distributed memory cache to the data source 117 of the storage apparatus 103.

  When executing the unload process, information stored in the load identification information 404 of the correspondence information 183 is designated. As a result, the loader unit 131 can identify the entry 191 to be unloaded. Furthermore, the loader unit 131 can identify the data source 117 to be unloaded with reference to the data source identification information 401 of the correspondence information 183.

  The loader unit 131 refers to the arrangement designation information 303 of the data source information 182 and determines whether or not the arrangement location of the designated data source 117 is designated (step S1501).

  For example, when “distributed area” or “shared area” is stored in the arrangement designation information 303, it is determined that the arrangement location of the designated data source 117 is designated. On the other hand, when “no designation” is stored in the arrangement designation information 303, it is determined that the arrangement location of the designated data source 117 is not designated.

  When it is determined that the location of the data source 117 is specified, the loader unit 131 determines whether or not the specified location is the shared area 172 (step S1502). Specifically, the loader unit 131 determines whether the “shared area” is stored in the arrangement designation information 303 of the data source information 182.

  When it is determined that the designated arrangement location is not the shared area 172, the loader unit 131 reflects the value 192 stored in the distributed area 171 in the corresponding data source 117 (step S1503). Specifically, the loader unit 131 writes the updated value 192 value to the corresponding data source 117. If the value 192 has not been updated, this process can be omitted.

  The loader unit 131 updates the reference number 402 of the correspondence information 183 (step S1504). Specifically, the loader unit 131 decrements the value of the reference number 402 by “1”. Thus, the current number of loads can be managed.

  Further, the loader unit 131 deletes the entry 191 from the distributed memory cache (step 1505) and ends the process. At this time, the value 192 stored in the distribution area 171 is also deleted.

  If it is determined in step S1502 that the designated placement location is the shared area 172, the loader unit 131 determines whether the reference number 402 is “1” or less (step S1509).

  If it is determined that the reference number 402 is equal to or less than “1”, the loader unit 131 proceeds to step S1503. If it is determined that the reference number 402 is greater than “1”, the loader unit 131 proceeds to step S1504.

  If it is determined in step S1501 that the arrangement location of the data source 117 is not specified, the loader unit 131 executes an arrangement location conversion process (steps S1506 to S1508).

  First, the loader unit 131 determines whether or not the reference number 402 is “1” or less (step S1506).

  When it is determined that the reference number 402 is “1” or less, all the entries 191 generated from the same data source 117 are deleted from the distributed memory cache by the unload processing. Therefore, the loader unit 131 proceeds to step S1503 to reflect the update of the value 192. In step S1504, the loader unit 131 notifies each host computer 102 of an instruction to delete the shared value management information 144 and the shared area management information 145.

  When it is determined that the reference number 402 is larger than “1”, the loader unit 131 determines whether or not the reference number 402 is “3” or more (step S1507).

  If it is determined that the reference number 402 is “3” or more, the loader unit 131 proceeds to step S1504 to delete the entry 191.

  When it is determined that the reference number 402 is smaller than “3”, that is, the reference number 402 is “2”, the loader unit 131 moves the shared value 192 from the shared area 172 to the distributed area 171 (step S1508). The process proceeds to step S1504. This is because, after the designated entry 191 is deleted, the shared value 192 is referred to only from one UAP 161, so it is not necessary to store it in the shared area 172.

  At this time, the loader unit 131 updates the entry 191 as the value 192 is transferred. That is, the entry 191 as shown in FIG. 12 or 13 is converted into the entry 191 as shown in FIG. Specifically, “distributed area” is stored in the area usage information 702, and the shared value management information pointer 705 is deleted. The storage location information 703 stores information indicating the storage location of the value 192 in the distribution area 171.

  In step S1504, the loader unit 131 notifies each host computer 102 of an instruction to delete the shared value management information 144 and the shared area management information 145.

  In the first embodiment, the host computer 101 is described as including the loader unit 131 and the like, but the present invention is not limited to this. For example, at least one host computer 102 may be connected to the storage apparatus 103, and may include a loader unit 131, a data source management unit 121, and the like.

  According to the first embodiment, data can be managed so that a plurality of applications can share value with respect to key / value type data (entries) generated from the same record by different applications. As a result, when a value is updated by one application without having to recreate an entry from the same data source, it can be reflected in an entry accessed by another application. Therefore, data consistency in the distributed memory cache can be maintained.

[Second Embodiment]
In the first embodiment, the shared area 172 is configured on the shared storage device 105 connected to the host computer 102. However, in the second embodiment, the shared area 172 is set on the memory 114 of each host computer 102. The point to be configured is different. Hereinafter, the second embodiment will be described focusing on differences from the first embodiment.

  FIG. 16 is a block diagram illustrating a configuration example of a computer system according to the second embodiment of this invention.

  Since the configuration of the host computer 101 is the same as that of the first embodiment, description thereof is omitted.

  The computer system according to the second embodiment does not include the shared storage device 105. In the second embodiment, the host computer 102 configures a shared area 1601 on the memory 114. Here, the information stored in the distributed area 171 and the shared area 1601 is the same as the information stored in the distributed area 171 and the shared area 172 in the first embodiment, respectively. It is more desirable to store key / value type data including shared value identification information 1203 and shared value 1104 of entry 191 shown in FIG. 13 in shared area 1601.

  In a framework for realizing key / value type data by using the shared value identification information 1203 as a key for uniquely identifying the shared value 192 in addition to the normal key 701, it is easier than the first embodiment. The shared value 192 can be managed.

  Key / value type data that does not require sharing of the value 192 is arranged in the distribution area 171, and key / value type data that requires sharing of the value 192 is arranged in the sharing area 1601. At this time, an entry having a normal key as the first-stage key is arranged in the distributed area 171, and an entry having the shared value identification information 1203 as the second-stage key is arranged in the shared area 1601. The sharing of the value 192 can be realized with the key structure. In other words, the value stored in the shared area 1601 can be configured similarly to the entry 191.

  In this embodiment, the distributed area 171 and the shared area 172 are configured separately, but the present invention is not limited to this, and the entry 191 and the value 192 may be managed in one area. That is, as long as the entry 191 as shown in FIGS. 12 and 13 can be generated, the shared value 192 may be stored in any storage area.

  Other configurations and processes are the same as those in the first embodiment, and thus description thereof is omitted.

  According to the second embodiment, by recognizing a value shared by a plurality of applications without using the shared storage device 105, one entry is not required to be recreated through the same data source. Updates to entries generated from applications can be reflected in entries accessed by other applications.

[Third Embodiment]
In the third embodiment, the distributed memory cache management unit 141 of the memory 114 of the host computer 102 acquires the access statistical information of the entry 191 in the distributed memory cache, and based on the information, the distribution area 171 and the shared area 172 It differs from the first and second embodiments in that it includes processing for determining the transition condition between the first and second embodiments.

  Hereinafter, the third embodiment will be described focusing on differences from the first embodiment.

  FIG. 17 is a block diagram showing a configuration example of a computer system according to the third embodiment of the present invention.

  The distributed memory cache management unit 141 is different from the first embodiment in that it includes access statistical information 1701 and migration condition information 1702.

  The access statistical information 1701 stores statistical information regarding access to the entry 191 in the distributed memory cache. The migration condition information 1702 stores information on a determination criterion for transferring the value 192 to the shared area 172 and a determination criterion for transferring the shared value 192 to the distributed area 171.

  Since other configurations are the same as those of the first embodiment, description thereof is omitted.

  FIG. 18 is an explanatory diagram illustrating a configuration example of the entry 191 according to the third embodiment of this invention.

  The shared value management information 144 of the third embodiment newly includes an access statistical information pointer 1801. The access statistical information pointer 1801 stores a pointer to the access statistical information 1701. By storing the access information for each entry 191 in the access statistical information 1701, the location of the value 192 for each entry can be changed.

  FIG. 19 is an explanatory diagram illustrating a configuration example of the access statistical information 1701 according to the third embodiment of this invention.

  The access statistical information 1701 includes an access count 1901 and an update count 1902.

  The access count 1901 stores the access count such as reference and update to the corresponding entry 191. The update count 1902 stores the access count regarding the update to the corresponding entry 191. In this embodiment, the access count 1901 and the update count 1902 are used as information stored in the access statistical information 1701, but the present invention is not limited to this.

  FIG. 20 is an explanatory diagram illustrating a configuration example of the migration condition information 1602 according to the third embodiment of this invention.

  The migration condition information 1602 includes a shared area migration condition 2001 and a distributed area migration condition 2002.

  The shared area migration condition 2001 stores a condition for migrating to the shared area 172 when the value 192 is stored in the distributed area 171.

  For example, the shared area migration condition 2001 stores a threshold value of the update count 1902 included in the access statistical information 1701. In this case, if the set threshold value is exceeded, the value 192 is transferred from the distributed area 171 to the shared area 172. Further, a threshold value of the ratio of the number of updates to the number of accesses may be stored in the shared area migration condition 2001.

  The distributed area migration condition 2002 stores a condition for migrating to the distributed area 171 when the value 192 is stored in the shared area 172.

  For example, the distribution area migration condition 2002 stores a threshold value of the ratio of the update count 1902 to the access count 1901 included in the access statistical information 1701. In this case, the shared value 192 is transferred from the shared area 172 to the distributed area 171 when it falls below the set threshold value.

  FIG. 21 is a flowchart for explaining data migration processing in the third embodiment of the present invention.

  The data migration processing is executed by the entry arrangement management unit 142 when an entry of key / value type data is loaded and when the entry is unloaded to the data source 117. Further, the data migration processing is not limited to entry loading and unloading, and may be executed between job executions during job net execution or periodically.

  The entry arrangement management unit 142 determines whether or not the value 192 of the target entry 191 is stored in the shared area 172 (step S2101). Specifically, the entry arrangement management unit 142 refers to the area usage information 702 of the entry 191 and determines whether or not the value 192 is stored in the shared area 172.

  When it is determined that the value is stored in the shared area 172, the entry arrangement management unit 142 refers to the entry 191, acquires the access statistical information 1701, and determines whether or not the shared area migration condition is satisfied ( Step S2102).

  Specifically, the entry arrangement management unit 142 acquires the access statistical information 1701 by referring to the access statistical information pointer 1801 of the entry 191. The entry arrangement management unit 142 refers to the migration condition information 1702 and the acquired 1701 and determines whether or not the shared area migration condition is satisfied. This is a determination process for preferentially using the currently stored shared area 172 if the shared area migration condition and the distributed area migration condition are satisfied at the same time.

  When it is determined that the shared area migration condition is satisfied, the entry arrangement management unit 142 ends the process.

  If it is determined that the shared area migration condition is not satisfied, the entry arrangement management unit 142 refers to the entry 191 to acquire the access statistical information 1701 and determines whether or not the distributed area migration condition is satisfied (step S2103). . The process in step S2103 is the same as the process in step S2102 and will not be described.

  If it is determined that the distribution area migration condition is not satisfied, the entry arrangement management unit 142 ends the process.

  When it is determined that the distribution area migration condition is satisfied, the entry arrangement management unit 142 migrates the value 192 from the shared area 172 to the distribution area 171 (step S2104). At this time, the value 192 is set in the entry 191 in the entry arrangement management unit 142. The value 192 may not be set in the entry 191. In this case, the entry arrangement management unit 142 may generate a pointer to the value 192 stored in the distribution area 171 and set the generated pointer in the entry arrangement management unit 142.

  The entry arrangement management unit 142 executes a reset process for the access statistical information 1701 (step S2105), and ends the process. Note that the reset processing of the access statistical information 1701 is not necessarily executed, and may be executed only when necessary.

  If it is determined in step S2101 that the value is not stored in the shared area 172, the entry arrangement management unit 142 refers to the entry 191, acquires the access statistical information 1701, and determines whether or not the distributed area migration condition is satisfied. Is determined (step S2106).

  Specifically, the entry arrangement management unit 142 acquires the access statistical information 1701 by referring to the access statistical information pointer 1801 of the entry 191. The entry arrangement management unit 142 refers to the migration condition information 1702 and the acquired access statistical information 1701 to determine whether or not the distributed region migration condition is satisfied. This is a determination process for preferentially using the currently stored distributed area 171 if both the distributed area transfer condition and the shared area transfer condition are satisfied at the same time.

  When it is determined that the distribution area migration condition is satisfied, the entry arrangement management unit 142 ends the process.

  If it is determined that the distribution area migration condition is not satisfied, the entry arrangement management unit 142 refers to the entry 191 to obtain the access statistical information 1701 and determines whether the shared area migration condition is satisfied (step S2107). . Since the process in step S2107 is the same as the process in step S2106, description thereof is omitted.

  If it is determined that the shared area migration condition is not satisfied, the entry arrangement management unit 142 ends the process.

  If it is determined that the shared area migration condition is satisfied, the entry arrangement management unit 142 migrates the value 192 from the distributed area 171 to the shared area 172 (step S2108), and the process proceeds to step S2105. At this time, the entry arrangement management unit 142 generates information for accessing the shared value 192 stored in the shared area 172, and sets the generated information in the storage location information 703 of the entry 191.

  With the above processing, when there are a plurality of entries 191 generated from records of the same data source, it is possible to place them in an optimal distributed memory cache according to the usage status of the entries 191.

  The following can be considered as a configuration for realizing entry migration in the third embodiment.

  22 and 23 are explanatory diagrams illustrating an example of a logical data configuration of the entry 191 according to the third embodiment of this invention.

  FIG. 22 shows the data structure of the entry 191 when there are two entries generated from records of the same data source 117 and no value is shared. FIG. 23 is an explanatory diagram showing the data structure of the entry 191 when there are two entries generated from records of the same data source and the value is shared.

  Note that the method for generating the entry 191 as shown in FIGS. 22 and 23 is the same as the load processing in the first embodiment, and thus the description thereof is omitted.

  According to the third embodiment, it is possible to automate the value transfer process between the distributed area and the shared area based on the access statistical information to the entry in the distributed memory cache. As a result, even if the location of the value is not set, the value can be placed in an appropriate storage area.

DESCRIPTION OF SYMBOLS 101 Host computer 102 Host computer 103 Storage apparatus 104 Network 105 Shared storage apparatus 121 Data source management part 131 Loader part 141 Distributed memory cache management part 142 Entry arrangement management part 143 Shared area access management part 144 Shared value management information 145 Shared area management information 161 Application program (UAP)
171 Distributed area 172 Shared area 181 Distributed memory cache configuration information 182 Data source information 183 Corresponding information 191 Entry 192 Value 1701 Access statistics information 1702 Migration condition information

Claims (15)

A plurality of first computers having a first processor, a first memory connected to the first processor, and a first network interface connected to the first processor, and storing data; Two processors, a second memory connected to the second processor, and a second network interface connected to the second processor, and data stored in each of the plurality of first computers A computer system comprising: a second computer for managing
The computer system includes a storage generated by integrating storage areas provided by the plurality of first computers,
The second calculator is
A loader processing unit that divides a file including a plurality of records and distributes and stores the divided data in which the search key and the value indicating the content of the record are associated with each other in the storage, and the file that stores the file Having a storage section;
Storing storage configuration information including information on the storage area constituting the storage, and correspondence information for managing the correspondence between the divided data stored in the storage and the file,
Each of the plurality of first computers is
An application that processes data in units of file data, and a storage management unit that manages the storage,
Storing divided data management information for managing the divided data;
The storage stores first divided information including first access information for accessing the value of the first file and a first search key, and the value of the first file,
The second calculator is
When receiving the split data placement request of the first file including the information of the second search key from the application, the second search key specified by the application is generated,
Determining whether the value of the first file is a shared value shared by a plurality of the applications;
When it is determined that the value of the first file is a shared value, the second divided data including the second search key and the first access information is stored in the storage. Computer system. The storage includes a first storage area managed by each of the plurality of first computers, and a second storage area shared by all the first computers,
The second calculator is
Storing the divided data in the first storage area;
When the value of the file is not a shared value, the value of the file that is the source of the divided data is stored in the first storage area,
2. The computer system according to claim 1, wherein when the value of the file is a shared value, the value of the file that is the source of the divided data is stored in the second storage area. The second calculator is
When the first divided data and the value of the first file are stored in the first storage area, and the value of the first file is determined to be a shared value, the first data Transferring the value of the file from the first storage area to the second storage area;
Generating second access information for accessing the value of the first file stored in the second storage area;
Updating the first access information of the first divided data to the second access information;
The computer system according to claim 2, wherein the second divided data including the second search key and the second access information is stored in the first storage area. Each of the plurality of first computers stores access statistical information for monitoring access to the value, and migration condition information regarding a migration condition of a storage location of the value of the file,
Each of the plurality of first computers is
When the value of the first file is stored in the first storage area, the access statistical information corresponding to the value of the first file is acquired,
Determining whether to migrate the value of the first file to the second storage area based on the migration condition information and the acquired access statistical information;
If it is determined to migrate the value of the first file to the second storage area, the value of the first file is migrated to the second storage area;
Generating third access information for accessing the value of the first file stored in the second storage area after migration, and storing the first access information of the first divided data in the first The computer system according to claim 3, wherein the access information is updated to three access information. Each of the plurality of first computers is
When the value of the first file is stored in the second storage area, the access statistical information corresponding to the value of the first file is acquired,
Determining whether to migrate the value of the first file to the first storage area based on the migration condition information and the acquired access statistical information;
When it is determined that the value of the first file is to be transferred to the first storage area, the value of the first file is transferred to the first storage area;
Generate fourth access information for accessing the value of the first file stored in the first storage area after the migration, and store the second access information of the first divided data in the first 5. The computer system according to claim 4, wherein the access information is updated to four access information. The second calculator is
When receiving a deletion request for the first divided data from the first application, it is determined whether the value of the first file is a shared value;
When it is determined that the value of the first file is not a shared value, the update result of the value of the first file is reflected in the first file stored in the file storage unit,
Deleting the first divided data and the first file value stored in the first storage area;
4. The computer system according to claim 3, wherein when it is determined that the value of the first file is a shared value, the first divided data is deleted from the first storage area. When it is determined that the value of the first file is a shared value, it is further determined whether the value of the first file is accessed only from one application,
If it is determined that the value of the first file is accessed only from one application, the value of the first file is transferred from the second storage area to the first storage area,
Fifth access information for accessing the value of the first file stored in the first storage area after migration is generated, and the second access information of the other divided data is generated as the fifth access information. The computer system according to claim 6, wherein the computer system is updated to access information.   Whether to determine whether the value of the first file is the shared value, whether the second storage area is designated as the storage destination of the value of the first file in the placement request The computer system according to claim 2, wherein the computer system is determined. The computer system includes an external storage device to which all the first computers can be connected,
The computer system according to claim 2, wherein the second storage area is configured on the external storage device. Access information for accessing the value stored in the second storage area is identification information of the shared value,
3. The computer system according to claim 2, wherein the divided data in which the shared value identification information and the shared value are associated with each other is stored in the second storage area. A plurality of first computers having a first processor, a first memory connected to the first processor, and a first network interface connected to the first processor, and storing data; Two processors, a second memory connected to the second processor, and a second network interface connected to the second processor, and data stored in each of the plurality of first computers A data management method in a computer system comprising: a second computer for managing
The computer system includes a storage generated by integrating storage areas provided by the plurality of first computers,
The second calculator is
A loader processing unit that divides a file including a plurality of records and distributes and stores the divided data in which the search key and the value indicating the content of the record are associated with each other in the storage, and the file that stores the file Having a storage section;
Storing storage configuration information including information on the storage area constituting the storage, and correspondence information for managing the correspondence between the divided data stored in the storage and the file,
Each of the plurality of first computers is
An application that processes data in units of file data, and a storage management unit that manages the storage,
Storing divided data management information for managing the divided data;
The storage stores first divided information including first access information for accessing the value of the first file and a first search key, and the value of the first file,
The method
When the second computer receives a request for arranging the divided data of the first file including the information of the second search key from the application, the second search key designated by the application is used. A first step of generating;
A second step in which the second computer determines whether the value of the first file is a shared value shared by a plurality of the applications;
When the second computer determines that the value of the first file is a shared value, the second divided data including the second search key and the first access information is stored in the storage. A third step of storing;
A data management method comprising: The storage includes a first storage area managed by each of the plurality of first computers, and a second storage area shared by all the first computers,
The divided data is stored in the first storage area,
If the value of the file is not a shared value, the value of the file that is the source of the divided data is stored in the first storage area by the second computer,
12. The value of the file that is the source of the divided data is stored in the second storage area by the second computer when the value of the file is a shared value. The data management method described. The third step includes
When the first divided data and the value of the first file are stored in the first storage area, and the value of the first file is determined to be a shared value, the first data Migrating the value of the file from the first storage area to the second storage area;
Generating second access information for accessing the value of the first file stored in the second storage area;
Updating the first access information of the first divided data to the second access information;
Storing the second divided data including the second search key and the second access information in the first storage area;
The data management method according to claim 12, further comprising: Each of the plurality of first computers stores access statistical information for monitoring access to the value, and migration condition information regarding a migration condition of a storage location of the value of the file,
The method further comprises:
Each of the plurality of first computers acquires the access statistical information corresponding to the value of the first file when the value of the first file is stored in the first storage area. Steps,
Each of the plurality of first computers determines whether to migrate the value of the first file to the second storage area based on the migration condition information and the acquired access statistical information And steps to
When each of the plurality of first computers is determined to migrate the value of the first file to the second storage area, the value of the first file is migrated to the second storage area And steps to
Each of the plurality of first computers generates third access information for accessing the value of the first file stored in the second storage area after the migration, and the first division Updating the first access information of the data to the third access information;
Including
Each of the plurality of first computers acquires the access statistical information corresponding to the value of the first file when the value of the first file is stored in the second storage area. Steps,
Each of the plurality of first computers determines whether to migrate the value of the first file to the first storage area based on the migration condition information and the acquired access statistical information And steps to
When each of the plurality of first computers is determined to migrate the value of the first file to the first storage area, the value of the first file is migrated to the first storage area. And steps to
Each of the plurality of first computers generates fourth access information for accessing the value of the first file stored in the first storage area after the migration, and the first division Updating the second access information of the data to the fourth access information;
The data management method according to claim 13, further comprising: The method further comprises:
Determining whether the value of the first file is a shared value when the second computer receives a request to delete the first divided data from the first application;
When the second computer determines that the value of the first file is not a shared value, the update result of the value of the first file is reflected in the first file stored in the file storage unit. And steps to
The second computer deleting the first divided data and the first file value stored in the first storage area;
The second computer determining whether or not the value of the first file is accessed from only one application when it is determined that the value of the first file is a shared value; ,
If the second computer determines that the value of the first file is accessed only from one application, the value of the first file is transferred from the second storage area to the first file. The steps to migrate to storage space
The second computer generates fifth access information for accessing the value of the first file stored in the first storage area after the migration, and the second computer of the other divided data Updating access information to the fifth access information;
The second computer deleting the first divided data from the first storage area;
The data management method according to claim 13, further comprising:

Images