WO2012090281A1 - Dispositif de traitement de données - Google Patents

Dispositif de traitement de données Download PDF

Info

Publication number
WO2012090281A1
WO2012090281A1 PCT/JP2010/073608 JP2010073608W WO2012090281A1 WO 2012090281 A1 WO2012090281 A1 WO 2012090281A1 JP 2010073608 W JP2010073608 W JP 2010073608W WO 2012090281 A1 WO2012090281 A1 WO 2012090281A1
Authority
WO
WIPO (PCT)
Prior art keywords
update
data
unit
buffer
linkage
Prior art date
Application number
PCT/JP2010/073608
Other languages
English (en)
Japanese (ja)
Inventor
仁志 楓
Original Assignee
三菱電機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機株式会社 filed Critical 三菱電機株式会社
Priority to JP2012550608A priority Critical patent/JPWO2012090281A1/ja
Priority to PCT/JP2010/073608 priority patent/WO2012090281A1/fr
Publication of WO2012090281A1 publication Critical patent/WO2012090281A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems

Definitions

  • the present invention relates to a database linkage technique for transferring data between a plurality of database tables.
  • Patent Document 1 discloses database replication between a cooperation source database and a cooperation destination database having the same number, type, and size of columns constituting a table.
  • Patent Document 1 Since database replication targeted by Patent Document 1 is data transfer between databases of the same specification, data linkage between a linkage source database and a linkage destination database having different numbers, types, and sizes of columns constituting a table is performed. There is a problem that it cannot be done. When transferring data between multiple linkage source databases and multiple linkage destination databases, the number, type, and size of the columns that make up the table differ for each database, and the frequency of transactions and the update within each transaction. The number of records to be played varies from database to database.
  • the present invention has been made in view of these points, and has as its main object to efficiently perform database linkage between a plurality of linkage source databases and a plurality of linkage destination databases.
  • the data processing apparatus A data processing apparatus that reflects the update in the linkage source table in the linkage destination table, An updated record receiving unit for receiving an updated record updated in the cooperation source table; A linkage destination table update unit for updating the linkage destination table in units of records; Using the updated record, data that reflects the update in the cooperation source table, and an update reflection data generation unit that generates, as update reflection data, data that is at least part of the record in the cooperation destination table; An updated record buffer generating unit for generating an updated record buffer for delivering an updated record from the updated record receiving unit to the update reflected data generating unit; An update reflection data buffer generation unit that generates an update reflection data buffer for delivering update reflection data from the update reflection data generation unit to the cooperation destination table update unit; The updated record receiver is Write the received updated record to the updated record buffer, The update reflected data generation unit Asynchronously with the reception of the updated record by the updated record receiving unit, the updated record stored in the updated record buffer is read, and the update reflected data is generated using the read updated record.
  • the cooperation destination table update unit The update reflected data stored in the update reflected data buffer is read asynchronously with the reception of the updated record by the updated record receiving unit and the generation of the update reflected data by the update reflected data generating unit, and the read update reflected It is characterized in that the cooperation destination table is updated using data.
  • an updated record buffer for writing updated records and an update reflected data buffer for writing updated reflected data are generated, and the updated reflected data generating unit is asynchronous with the updated record receiving unit.
  • the linkage destination table update unit reads the update reflected data asynchronously with the updated record reception unit and the update reflected data generation unit. Since the table is updated, efficient pipeline processing is possible in database linkage between a plurality of linkage source databases and a plurality of linkage destination databases.
  • FIG. 1 is a diagram showing a configuration example of a database cooperation system according to Embodiments 1 to 4.
  • FIG. FIG. 3 is a diagram illustrating a configuration example of a database cooperation device according to the first embodiment.
  • FIG. 6 is a diagram illustrating an example of cooperation table information according to the first embodiment.
  • FIG. 4 is a diagram illustrating an example of cooperation management information according to the first embodiment.
  • FIG. 6 is a diagram illustrating an example of cooperative connection information according to the first embodiment. The figure which shows the example of the update data arrival log which concerns on Embodiment 1.
  • FIG. FIG. 3 is a diagram illustrating an example of a data configuration in a data processing control unit according to the first embodiment.
  • FIG. 4 shows an example of update data information according to the first embodiment.
  • FIG. 3 is a flowchart showing an operation example of the database linkage apparatus according to the first embodiment.
  • FIG. 3 is a flowchart showing an operation example of the database linkage apparatus according to the first embodiment.
  • the flowchart figure which shows the operation example of the database cooperation apparatus which concerns on Embodiment 2.
  • FIG. The figure which shows the structural example of the database cooperation apparatus which concerns on Embodiment 3, and a data evacuation apparatus.
  • FIG. 3 is a flowchart showing an operation example of the database linkage apparatus according to the first embodiment.
  • FIG. 5 is a diagram illustrating an example of a hardware configuration of the database linkage apparatus according to the first to fourth embodiments.
  • Embodiment 1 In the database (hereinafter referred to as DB) linkage technology, data changed in the linkage source database (linkage source table) is extracted, and the extracted data is converted into the format of the linkage destination database (linkage destination table). The converted data is reflected in the linked database. Examples of data conversion include the following. Conversion between uppercase and lowercase letters (abc ⁇ ABC): Conversion from lowercase letters to uppercase Conversion of numeric data units (1000 ⁇ 1): Conversion to K units Implementation of padding and change of the number of digits (123456 ⁇ 000123456): 6 digits From 9 to 9 digits
  • the first method is a method of transferring data from one database serving as a cooperation source to one database serving as a cooperation destination, like an ETL (Extract Transform Load) tool.
  • the second method is a method of transferring data from a plurality of cooperation source databases to a plurality of cooperation destination databases in a Hub type configuration in which a plurality of cooperation source databases and a plurality of cooperation destination databases are connected.
  • a second method that is, a database linkage method using a Hub type configuration will be described.
  • the hub-type database cooperation device receives a change data extracted in the cooperation source DB, a conversion process for matching the extracted data to the specification of the cooperation destination DB, and a CDW ( (Change Data Writer) processing.
  • a CDW Change Data Writer
  • the conversion unit and the CDW processing unit that perform pipeline processing when performing database cooperation and the queue configuration between tables is uniform, in other words, when the conversion processing and the CDW processing are synchronized, the database cooperation
  • Embodiments 1 to 4 are mainly intended to solve this problem, and in order to improve the throughput of the database linkage apparatus, a configuration in which the conversion unit and the CDW processing unit operate asynchronously in pipeline processing. explain.
  • a database linkage apparatus according to an embodiment will be described with reference to the drawings.
  • FIG. 1 shows a database linkage system according to the present embodiment, and shows a relationship among the database linkage apparatus 1, the linkage source system 2, and the linkage destination system 4.
  • the database linkage apparatus 1 is an example of a data processing apparatus. An example of the internal configuration of the database linkage apparatus 1 will be described later.
  • the database cooperation device 1 is connected to a plurality of cooperation source systems 2 and a plurality of cooperation destination systems 4. In FIG. 1, there are two cooperation source systems 2, that is, a cooperation source system A 2 a and a cooperation source system B 2 b. Further, the cooperation destination system 4 includes the cooperation destination system X4x, the cooperation destination system Y4y, and the cooperation destination system Z4z.
  • Each cooperation source system 2 includes a cooperation source database 3 and a CDC (Change Data Capture) 22.
  • the cooperation source database 3 includes one or more cooperation source tables.
  • the collaboration source database A3a includes two collaboration source tables, and the respective collaboration source tables are referred to as a collaboration source table A-1 and a collaboration source table A-2.
  • the collaboration source database B3b also includes two collaboration source tables, the respective collaboration source tables are referred to as a collaboration source table B-1 and a collaboration source table B-2.
  • the CDC 22 monitors an update log of the cooperation source database 3, detects an update of the cooperation source database 3, and extracts a record updated in the cooperation source database 3.
  • the CDC 22 is realized by a known technique.
  • a record extracted by the CDC 22 is also referred to as an updated record.
  • Each cooperation destination system 4 includes a cooperation destination database 5.
  • the cooperation destination database 5 includes one or more cooperation destination tables.
  • the cooperation destination database X5x includes two cooperation destination tables, and the respective cooperation destination tables are referred to as a cooperation destination table X-1 and a cooperation source table X-2.
  • the respective cooperation destination tables are expressed as a cooperation destination table Y-1 and a cooperation source table Y-2.
  • the cooperation destination database Z5z also includes two cooperation destination tables, the respective cooperation destination tables are referred to as a cooperation destination table Z-1 and a cooperation source table Z-2.
  • the database linkage apparatus 1 reflects the update in any linkage source database in any linkage destination table.
  • the database cooperation apparatus 1 can perform arbitrary database cooperation. For example, the update of one linkage source table is performed such that the updated record of the linkage source table A-1 is reflected in each of the linkage destination table X-1, the linkage destination table Y-1, and the linkage destination table Z-1.
  • the completed record may be reflected in two or more cooperation destination tables.
  • a combination of a part of the updated record of the cooperation source table A-1, a part of the updated record of the cooperation source table B-1 and a part of the updated record of the cooperation source table B-2 A combination of updated records in a plurality of linkage source tables may be reflected in one linkage destination table as reflected in the table Z-2.
  • the schema configuration of the linkage source database 3 and the linkage destination database 5 may be different, more specifically, the schema, table, and column configuration configured in the database instance.
  • the type, size, and stored data may be different from the column type, size, and stored data of the link destination database 5.
  • the number of cooperation source systems 2, the number of cooperation destination systems 4, the number of cooperation source databases 3, the number of cooperation destination databases 5, the number of cooperation source tables, and the number of cooperation destination tables shown in the present embodiment are examples. Yes, the number of each can be set arbitrarily.
  • FIG. 2 shows a configuration example of the database linkage apparatus 1 according to the present embodiment.
  • the configurations of the individual exclusive control unit 20 and the individual data storage unit 21 in the data storage unit 19 based on the data linkage definition by the reception unit 11, the conversion unit 12, and the CDW processing unit 13. And the configuration of the data storage unit 19 is managed by the cooperation management information 17.
  • the reception unit 11 receives the updated record detected by the CDC 22 in the collaboration source system 2 from the collaboration source table of the collaboration source database 3.
  • the receiving unit 11 is an example of an updated record receiving unit.
  • the receiving unit 11 is prepared for each cooperation source database 3. In FIG. 2, only one receiving unit 11 is shown for the sake of simplification. However, in light of the configuration in FIG. 1, the database linkage apparatus 1 in FIG. 2 receives updated records from the linkage source database A 3 a.
  • a receiving unit hereinafter referred to as a receiving unit 11a
  • a receiving unit 11b for receiving updated records from the cooperation source database B3b are provided.
  • the conversion unit 12 converts the data extracted from the cooperation source database 3 into the data specification of the cooperation destination database 5 when the database definition specifications of the cooperation source database 3 and the cooperation destination database 5 are different. For example, in the case of character string data, it has functions such as conversion from upper case to lower case and numerical operation such as digit alignment, and the conversion process is performed by a known technique.
  • the conversion unit 12 determines whether or not the input updated record needs to be converted, determines what conversion is necessary if conversion is necessary, converts the updated record if conversion is necessary, and after conversion Are updated in the individual data storage unit 21 described later. On the other hand, if there is no need for conversion, an unconverted updated record is written in the individual data storage unit 21.
  • the data that the conversion unit 12 writes to the individual data storage unit 21 reflects the update in the cooperation source table, whether it is a post-conversion updated record or a non-conversion updated record, and records in the cooperation destination table This is data (update reflected data) that becomes at least a part of.
  • the conversion unit 12 generates the update reflected data by converting the updated record if conversion is necessary, and generates the update reflected data without converting the updated record if conversion is not necessary. Yes.
  • the conversion unit 12 functions as an example of the update reflected data generation unit.
  • the conversion unit 12 is prepared for each cooperation source table. In FIG. 2, only one conversion unit 12 is shown for simplification of the drawing. However, in light of the configuration of FIG. 1, the database linkage device 1 of FIG.
  • conversion unit 12a-1 corresponds to the linkage source table A-1.
  • conversion unit 12a-2 corresponds to the conversion unit corresponding to cooperation source table A-2 (hereinafter referred to as conversion unit 12a-2), and the cooperation source table B-1
  • conversion unit 12b-1 A conversion unit (hereinafter referred to as conversion unit 12b-1) and a conversion unit (hereinafter referred to as conversion unit 12b-2) corresponding to cooperation source table B-2 are provided.
  • the CDW processing unit 13 has a function of reflecting the update reflection data generated by the conversion unit 12 in the cooperation destination database 5. More specifically, the update reflected data is reflected in the cooperation destination database 5 in units of records by using an ODBC (Open DataBase Connectivity) or the like. At this time, the CDW processing unit 13 collectively acquires a plurality of update reflection data from the conversion unit 12 and reflects them in the cooperation destination database 5. That is, the CDW processing unit 13 updates two or more records in the cooperation destination table in one transaction using the acquired plurality of update reflection data.
  • the CDW processing unit 13 is an example of a cooperation destination table update unit. The CDW processing unit 13 is prepared for each cooperation destination database 5. In FIG. 2, only one CDW processing unit 13 is shown for the sake of simplification.
  • the database linkage apparatus 1 in FIG. 2 has a CDW corresponding to the linkage destination database X5x.
  • a processing unit hereinafter referred to as CDW processing unit 13x
  • a CDW processing unit hereinafter referred to as CDW processing unit 13y
  • a CDW processing unit hereinafter referred to as CDW processing unit 13y
  • CDW processing unit 13z corresponding to cooperation destination database Z5z.
  • the linkage memory management unit 18 is a buffer area for transferring the updated record from the reception unit 11 to the conversion unit 12 on the main storage device (shared memory) of the database linkage device 1.
  • the individual data storage unit 21 is generated, and the individual data storage unit 21 which is a buffer area for transferring update reflected data from the conversion unit 12 to the CDW processing unit 13 is generated.
  • the individual data storage unit 21 for transferring the updated record from the receiving unit 11 to the conversion unit 12 is an example of an updated record buffer, and an individual data for transferring update reflected data from the conversion unit 12 to the CDW processing unit 13
  • the data storage unit 21 is an example of an update reflected data buffer.
  • the linkage memory management unit 18 is an example of an updated record buffer generation unit and an update reflection data buffer generation unit.
  • the conversion unit 12 includes four conversion units 12a-1, 12a-2, 12b-1, and 12b-2. 18 generates four individual data storage units 21 corresponding to the respective conversion units.
  • the cooperation memory management unit 18 includes the individual data storage unit 21 for the reception unit 11a to transfer the updated record of the cooperation source table A-1 to the conversion unit 12a-1, and the reception unit 11a for the cooperation source table A-2.
  • the receiving unit 11b generates four individual data storage units 21 for transferring the updated records of the cooperation source table B-2 to the conversion unit 12b-2.
  • the individual data storage unit 21 is generated.
  • the individual data storage unit 21 of the CDW processing unit 13 is generated for each conversion unit 12 that is the source of update reflected data.
  • the CDW processing unit 13x receives update reflection data from each of the conversion unit 12a-1, the conversion unit 12a-2, and the conversion unit 12b-1
  • individual data storage in which only the conversion unit 12a-1 can write update reflection data The individual data storage unit 21 in which only the unit 21 and the conversion unit 12a-2 can write update reflected data, and the individual data storage unit 21 in which only the conversion unit 12b-1 can write update reflected data are generated.
  • the individual exclusive control unit 20 provided for each individual data storage unit 21 performs exclusive control for writing updated records or update reflected data to each individual data storage unit 21.
  • the individual exclusive control unit 20 is an example of an exclusive control unit.
  • a logical address space in which one or more sets of the individual data storage unit 21 and the individual exclusive control unit 20 are stored is referred to as a data storage unit 19.
  • the logical address space in which the pair of the individual data storage unit 21 and the individual exclusive control unit 20 for the conversion unit 12b-1 is stored becomes the data storage unit 19 of the CDW processing unit 13x.
  • the data processing control unit 16 is a queue that stores messages for interprocess communication. That is, each receiving unit 11 registers a message notifying the writing of the updated record in the data processing control unit 16 after writing the updated record in the individual data storage unit 21.
  • Each conversion unit 12 calls a message from the data processing control unit 16 and reads an updated record from the corresponding individual data storage unit 21.
  • Each conversion unit 12 writes the update reflected data in the individual data storage unit 21 and then registers a message informing the writing of the update reflected data in the data processing control unit 16.
  • Each CDW processing unit 13 calls a message from the data processing control unit 16 and reads the update reflected data from the corresponding individual data storage unit 21.
  • the update data arrival log 15 is information indicating the reception status of updated records by the receiving unit 11, and more specifically, information indicating the number of received records per unit time for each cooperation source table. Details of the update data arrival log 15 will be described later.
  • the data linkage design unit 6 defines a linkage relationship between the linkage source database 3 and the linkage destination database 5.
  • the definition of the linkage relationship indicates the corresponding relationship between the schema, table, and column provided in the linkage source database 3 and the schema, table, and column provided in the linkage destination database 5.
  • the data linkage design unit 6 may be a tool equipped with a GUI (Graphical User Interface).
  • the data linkage design unit 6 may operate on hardware different from that of the database linkage device 1.
  • the data linkage design unit 6 outputs the database linkage definition as data linkage design information 7 and uses it as input data of the integrated linkage information 8. Details of the data linkage design information 7 and the integrated linkage information 8 will be described later.
  • the cooperative memory management unit 18 calculates the size required for the individual data storage unit 21 for each conversion unit 12. More specifically, the cooperation memory management unit 18 obtains, for each conversion unit 12, the number of received records per unit time from the target cooperation source table and the record length of the target cooperation source table. By integrating, the size required for the conversion unit 12 is calculated. Then, the cooperative memory management unit 18 assigns the individual data storage unit 21 corresponding to the calculated size to the conversion unit 12. Note that the number of received records per unit time from the linkage source table can be derived from the update data arrival log 15, and the record length of the linkage source table can be derived from the linkage source table information 9 or the like.
  • the cooperative memory management unit 18 calculates the capacity required for the individual data storage unit 21 for each CDW processing unit 13. As described above, the linked memory management unit 18 generates the individual data storage unit 21 for each conversion unit 12 that is the update reflection data transfer source for each CDW processing unit 13. Calculate the required capacity in 12 units. More specifically, the cooperative memory management unit 18 determines the data length of the update reflected data transferred from the conversion unit 12 by the CDW processing unit 13 for each conversion unit 12 from which the update reflected data is transferred, and the conversion unit. The number of received records per unit time from 12 linkage source tables and the number of records to which the CDW processing unit 13 reflects the update reflected data are integrated, and the individual data storage unit 21 for the capacity obtained by the integration. Secure.
  • the receiving unit 11 receives the updated record transmitted from the CDC 22 and writes the updated record in the data storage unit 19 assigned to the conversion unit 12 designated as the transfer destination of the received updated record.
  • the receiving unit 11 exists for each cooperation source database 3, and the conversion unit 12 exists for each cooperation source table.
  • the receiving unit 11a receives the updated record of the cooperation source table A-1 and the updated record of the cooperation source table A-1 of the cooperation source database 3.
  • the reception unit 11a writes the received updated record to the individual data storage unit 21 for the conversion unit 12a-1, and updates the cooperation source table A-2.
  • the receiving unit 11a writes the received updated record in the individual data storage unit 21 for the conversion unit 12a-2.
  • the conversion unit 12 reads the updated record from the assigned individual data storage unit 21, generates the update reflection data using the read updated record, and the CDW designated as the delivery destination of the generated update reflection data Write to the individual data storage unit 21 of the processing unit 13.
  • the transfer destination of the conversion unit 12a-1 is the CDW processing unit 13x
  • the CDW processing unit 13x receives update reflected data from each of the conversion unit 12a-1 and the conversion unit 12b-1
  • the CDW processing unit 13x In contrast, an individual data storage unit 21 for the conversion unit 12a-1 and an individual data storage unit 21 for the conversion unit 12b-1 are generated. Therefore, the conversion unit 12a-1 writes the update reflected data in the individual data storage unit 21 for the conversion unit 12a-1 of the CDW processing unit 13x.
  • the transfer destination of the conversion unit 12a-1 is the CDW processing unit 13x and the CDW processing unit 13y
  • the read updated record is converted in accordance with the cooperation destination database of the CDW processing unit 13x to perform the CDW processing.
  • the update reflection data for the unit 13x is generated, and the update reflection data for the CDW processing unit 13y is generated by performing conversion according to the cooperation destination database of the CDW processing unit 13y.
  • the update reflection data for the CDW processing unit 13x is written in the individual data storage unit 21 for the conversion unit 12a-1 of the CDW processing unit 13x, and the update reflection data for the CDW processing unit 13y is written in the conversion unit 12a of the CDW processing unit 13y. Write to the individual data storage unit 21 for -1.
  • the conversion unit 12a-1 performs writing to the individual data storage unit 21 of the CDW processing unit 13x and writing to the individual data storage unit 21 of the CDW processing unit 13y in parallel.
  • reading of updated records, generation of update reflected data, and writing of update reflected data to the individual data storage unit 21 in each conversion unit 12 are asynchronous with reception of updated records by the receiving unit 11 and writing of updated records. To be done.
  • the receiving unit 11 receives the updated record, writes the updated record, and each converting unit 12 updates the updated record. Reading, generation of update reflection data, and writing of update reflection data to the individual data storage unit 21 are performed asynchronously. For this reason, each CDW processing unit 13 can update two or more records in the cooperation destination table in one transaction. In addition, each CDW processing unit 13 reads the update reflected data and updates the cooperation destination table in parallel with the reading of the update reflection data and the update of the cooperation destination table in the other CDW processing unit 13.
  • the data linkage design information 7 may be described in a format such as XML (Extensible Markup Language), or may be a format that is externally taken into the database linkage device 1 in the form of a file.
  • the data linkage design information 7 includes linkage source table information 9, linkage destination table information 10, and linkage connection information 24. 2 shows a state in which the cooperation source table information 9, the cooperation destination table information 10, and the cooperation connection information 24 included in the data cooperation design information 7 are expanded in the memory of the database cooperation device 1.
  • the data linkage design information 7 is a file described in the XML format, it is possible to read the contents of the database linkage design information 7 from the file each time the data linkage processing unit 23 performs the database linkage processing. Not right. For this reason, in this embodiment, the data linkage design information 7 is received as an input and stored as the integrated linkage information 8 in the memory of the database linkage device 1.
  • the linkage source table information 9 and the linkage destination table information 10 are composed of table definition information 201 and column definition information 202 as shown in FIG.
  • the CDC / CDW identifier is an identifier that identifies the CDC 22 or the CDW processing unit 13.
  • the table identifier is an identifier for uniquely identifying the cooperation source table or the cooperation destination table in the database cooperation device 1.
  • the table type is information indicating the type of the cooperation source table or the cooperation destination table
  • the table name is information indicating the name of the cooperation source table or the cooperation destination table.
  • the table definition information 201 includes information such as the number of columns of the linkage source table or linkage destination table, column information serving as a primary key, table information having a reference relationship, definition registration date, update date, and definition version. Is included.
  • the column identifier is an identifier for uniquely identifying the column within the database linkage apparatus 1.
  • the table identifier is an identifier of the table to which the column belongs.
  • the data type indicates the data type of the column, and the numerical accuracy indicates the numerical accuracy (the number of digits, the number of significant digits less than a decimal) when the column data type is a numerical type.
  • the column definition information 202 includes information such as the data length of the data included in the column, the column name, and the parent column identifier in the case of a foreign key.
  • the cooperation management information 17 includes a process management table 203, data storage unit information 204, individual exclusive control information 205, individual data storage information 206, external identifier information 207, and open handle information 208.
  • the process management table 203 includes a process ID, a process type, an external identifier, a state, a data processing control unit ID, a read control flag, a data storage unit ID, an initialization handle, and an open handle.
  • the process ID is an identifier of each process of the reception unit 11, the conversion unit 12, and the CDW processing unit 13.
  • the process type indicates whether the process specified by the process ID is a process of the reception unit 11, the conversion unit 12, or the CDW processing unit 13.
  • the data processing control unit ID is an identifier of the data processing control unit 16 and indicates whether it is a message queue from the reception unit 11 to the conversion unit 12 or a message queue from the conversion unit 12 to the CDW processing unit 13. Show.
  • the read control flag indicates whether data can be read from the individual data storage unit 21.
  • the data storage unit information 204 includes a data storage unit ID, individual data count, individual exclusive control information, connection management ID, cooperation target logical table ID, and individual data storage information.
  • the data storage unit ID is an identifier of the data storage unit 19.
  • the connection management ID is an identifier set for each combination of the cooperation source table and the cooperation destination database 5. In the above-described example, there are four linkage source tables (A-1, A-2, B-1, B-2) and three linkage destination tables (X, Y, Z). Connection management IDs are provided.
  • the cooperation target logical table ID indicates the identifier of the cooperation source table when the process identified by the process ID of the process management table 203 is the reception unit 11 and the conversion unit 12, and indicates the cooperation destination table when the process is the CDW processing unit 13. An identifier is shown.
  • the individual exclusive control information 205 includes an individual exclusive control ID, a buffer exclusive control flag, and an overflow control flag.
  • the individual exclusive control ID is an identifier of the individual exclusive control unit 20.
  • the individual data storage information 206 includes an individual data storage unit ID, a plurality of connection management ID duplications, and a total column length (Sl) to be linked.
  • the individual data storage unit ID is an identifier of the individual data storage unit 21.
  • the connection management ID duplication indicates how many records of the cooperation destination table are updated by the update reflected data read from the individual data storage unit 21 specified by the individual data storage unit ID. For example, when the CDW processing unit 13x reflects one update reflected data from the conversion unit 12a-1 in the cooperation destination table X-1 and the cooperation destination table X-2 of the cooperation destination database 5x, the conversion unit 12a-1 In the individual data storage unit 21 that delivers the update reflected data to the CDW processing unit 13x, the connection management ID duplication number is 2.
  • the cooperation target column length total number (Sl) indicates the data length (record length) written in the individual data storage unit 21 specified by the individual data storage unit ID. For example, in the case of the individual data storage unit 21 from the conversion unit 12 to the CDW processing unit 13, the data length (record length) of the update reflected data written in the individual data storage unit 21 by the writing source conversion unit 12 is indicated.
  • the external identifier information 207 indicates the CDCid when the process specified by the process ID of the process management table 203 is a receiving unit, the cooperation source logical table ID when the process is a conversion unit, and the cooperation destination database ID when the process is a CDW processing unit.
  • the open handle information 208 includes an open handle, input / output edges, exclusive control information (semaphore ID, etc.), and individual data storage information (address to memory, etc.).
  • the input edge is a data input source
  • the output edge is a data output destination.
  • the CDC is an input edge
  • the individual data storage unit 21 for delivering the updated record to the conversion unit 12 is an output edge.
  • the individual data storage unit 21 for receiving updated records serves as an input edge
  • the individual data storage unit 21 for delivering update reflected data to the CDW processing unit 13 serves as an output edge.
  • the individual data storage unit 21 for receiving the update reflected data is an input edge
  • the cooperation destination table is an output edge.
  • the CDW processing unit 13x receives update reflected data from each of the converting unit 12a-1 and the converting unit 12b-1, the individual data storage unit 21 for the converting unit 12a-1 and the converting unit 12b-1 The individual data storage unit 21 becomes an input edge.
  • the CDW processing unit 13x updates the cooperation destination table X-1 and the cooperation destination table X-2, the cooperation destination table X-1 and the cooperation destination table X-2 are output edges, respectively.
  • connection input information 210 connection input information 210
  • connection output information 211 connection output information 211
  • conversion unit connection information 212 conversion unit connection information 212
  • the connection input information 210 is information about the receiving unit 11 and includes a connection management ID, a connection management version, an input edge ID, an output edge number, a cooperation source logical table ID, and a conversion process identifier.
  • the conversion process identifier is the process ID of the conversion unit 12 to which the receiving unit 11 delivers the updated record.
  • the connection output information 211 is information about the CDW processing unit 13 and includes a connection management ID, a connection management version, an input edge ID, a CDWid, an output edge ID, runtime additional information (record length), and a cooperation destination logical table ID. .
  • the additional information (record length) at the time of execution is the data length (record length) of the update reflected data transferred from the conversion unit 12 that is the transfer source by the CDW processing unit 13.
  • the conversion unit connection information 212 is information about the conversion unit 12 and includes a process identifier, the number of output edges, output edge information, and an input edge ID.
  • the output edge information 213 includes an output edge ID and additional information at the time of execution. It consists of a pair.
  • the update data arrival log 15 has a configuration as shown in FIG.
  • the update data arrival log 15 includes an ID for uniquely identifying a record, a linkage source logical table ID, a connection management ID, an average number of transactions per unit time, an average number of records per transaction (number of updated records), and a final update. Consists of days.
  • the conversion unit 12 and the CDW processing unit 13 call the cooperative memory management unit 18 as an initialization process when generating a process, and generate an individual data storage unit 21 for acquiring data from other processing units.
  • the processing flow for calculating the size of the individual data storage unit 21 based on FIG. 10 will be described below.
  • the CDW processing unit 13 notifies the associated memory management unit 18 of the CDWid that is an identifier of itself (CDW processing unit 13) (S201).
  • the linked memory management unit 18 extracts one or more connection output information 211 (FIG. 5) of the linked connection information 24 indicating the same CDWid as the notified CDWid, and classifies the extracted connection output information 211 for each connection management ID. (S202). For example, when the CDW processing unit 13x updates the cooperation destination table of the cooperation destination database 5x using the update reflected data from each of the conversion unit 12a-1 and the conversion unit 12b-1, it cooperates with the cooperation source table A-1.
  • Connection output information 211 in which a connection management ID corresponding to a combination with the destination database 5x is described, and a connection output in which a connection management ID corresponding to a combination of the cooperation source table B-1 and the cooperation destination database 5x is described.
  • Information 211 exists.
  • the cooperative memory management unit 18 groups the connection output information 211 for each connection management ID in S202.
  • the cooperative memory management unit 18 acquires the additional information at run time (record length) from the connection output information 211, and calculates the sum of the acquired additional information at run time (record length). Then, the sum of the additional information (record) at the time of execution is defined as the size (S1) of the record in the block of the individual data storage unit 21 (S203, S204). Since the connection management ID is an identifier that identifies a combination of the cooperation source table and the cooperation destination database, the output edge ID may be different even if the connection output information 211 has the same connection management ID.
  • connection management ID is Even if they are the same, there is connection output information 211 in which the output edge ID corresponding to the cooperation destination table X-1 is described, and connection output information 211 in which the output edge ID corresponding to the cooperation destination table X-2 is described. Will do.
  • the cooperation memory management unit 18 obtains the sum of the additional information (record length) at the time of execution for each group of the connection output information 211.
  • the cooperative memory management unit 18 obtains the average transaction count / unit time (Stn) and average record count / transaction (Stl) values from the update data arrival log 15 (FIG. 6). (S205). Next, the cooperative memory management unit 18 acquires a buffer having a size corresponding to (Stn ⁇ connection management ID duplication number ⁇ Stl ⁇ Sl) as the individual data storage unit 21 (S206).
  • the plurality of connection management IDs used for the integration in S206 are grouped as having the same connection management ID as a result of the number of connection output information 211 having the same connection management ID, that is, the classification in S202. This is the number of connection output information 211.
  • the cooperative memory management unit 18 stores the connector (for example, memory address) to the individual data storage unit 21 generated in S206 in the individual data storage unit ID column of the individual data storage information 206 (FIG. 4).
  • the connection management ID duplication number used in the accumulation in S206 is stored in the connection management ID duplication number column.
  • the linked memory management unit 18 performs the processing from S203 onward for all connection management IDs belonging to the list of linked destination logical table IDs (S207). After the processing from S203 is performed on all connection management IDs, the cooperative memory management unit 18 generates an initialization handle for the generated individual data storage unit 21, and uses the generated initialization handle as a CDW. Notify the processing unit 13.
  • the cooperative memory management unit 18 generates an individual data storage unit 21 for each connection management ID for each CDW processing unit 13.
  • the connection management ID is different for each cooperation source table, and the conversion unit 12 is provided for each cooperation source table.
  • the linked memory management unit 18 generates an individual data storage unit 21 for each conversion unit 12 that is the source of update reflected data for each CDW processing unit 13.
  • the size of the individual data storage unit 21 is the unit of the updated record from the linkage source table targeted by the conversion unit 12 and the data length (Sl) of the update reflected data transferred from the conversion unit 12 to the CDW processing unit 13.
  • the size is obtained by integrating the number of receptions per time (Stn ⁇ Stl) and the number of records (multiple connection management IDs) that the CDW processing unit 13 reflects the update reflection data.
  • the conversion unit 12 acquires the input edge ID from the conversion unit connection information 212 (FIG. 5) in which the ID of itself (the conversion unit 12) is described in the process identifier column, and notifies the cooperation memory management unit 18 of the input edge ID.
  • the cooperation memory management unit 18 extracts the connection input information 210 (FIG. 5) in which the same input edge ID as the notified input edge ID is described, and acquires the cooperation source logical table ID in the extracted connection input information 210. To do.
  • the cooperation memory management unit 18 extracts the cooperation source table information 9 (table definition information: FIG. 3) in which the same cooperation source logical table ID as the acquired cooperation source logical table ID is included in the table identifier column.
  • the column information in the extracted cooperation source table information 9 is acquired.
  • the cooperative memory management unit 18 acquires the data length for each column identifier of the column definition information 202 (FIG. 3) associated with the acquired column information, and totals the acquired data length.
  • the cooperation memory management unit 18 defines the total value as the total record target column length as the in-block record size (Sl) of the individual data storage unit.
  • the linkage memory management unit 18 uses the linkage source table ID as a key to obtain the average transaction count / unit time (Stn) and average record count / transaction (Stl) values from the update data arrival log 15 (FIG. 6). get.
  • the cooperative memory management unit 18 acquires a buffer having a size corresponding to Stn ⁇ St1 ⁇ S1 as the individual data storage unit 21.
  • the cooperation memory management unit 18 stores the generated connector (for example, memory address) to the individual data storage unit 21 in the individual data storage unit ID column of the individual data storage information 206 (FIG. 4), and The original logical table ID is stored in the cooperation target logical table ID column of the data storage unit information 204 (FIG. 4). Finally, the cooperative memory management unit 18 generates an initialization handle and notifies the conversion unit 12 of the generated initialization handle.
  • the generated connector for example, memory address
  • the receiving unit 11 acquires an open handle for writing the updated record acquired from the CDC 22 to the individual data storage unit 21 of the conversion unit 12.
  • the receiving unit 11 notifies the cooperation memory management unit 18 of the initialization handle and the input edge ID.
  • the cooperation memory management unit 18 extracts the connection input information 210 (FIG. 5) of the cooperation connection information in which the same input edge ID as the notified input edge ID is indicated, and the conversion process identifier in the extracted connection input information 210 And obtain the cooperation source logical table ID.
  • the cooperation memory management unit 18 extracts the process management table 203 (FIG. 4) of the cooperation management information 17 indicating the same process ID as the acquired conversion process identifier, and the data in the extracted process management table 203 Acquires the storage unit ID.
  • the data storage unit information 204 associated with the acquired data storage unit ID is extracted from the data storage unit information 204 associated with the acquired data storage unit ID.
  • the cooperative memory management unit 18 acquires the individual data storage unit ID stored in the individual data storage information in the extracted data storage unit information 204.
  • the cooperative memory management unit 18 maps the shared memory using the acquired individual data storage unit ID as a key, and sets the individual exclusive control ID stored in the individual exclusive control information in the extracted data storage unit information 204 as a key. Create a semaphore for. Further, the process management table 203 (FIG.
  • the conversion unit 12 acquires an open handle for writing the update reflected data into the individual data storage unit 21.
  • the conversion unit 12 notifies the cooperation memory management unit 18 of the initialization handle and the output edge ID.
  • the cooperative memory management unit 18 extracts the connection output information 211 (FIG. 5) of the cooperative connection information 24 in which the same output edge ID as the notified output edge ID is indicated, and connects to the CDWid from the extracted connection output information 211. Get the management ID.
  • the cooperation memory management unit 18 extracts the process management table 203 (FIG. 4) of the cooperation management information 17 in which the same process ID as the acquired CDWid is shown, and is associated with the extracted process management table 203.
  • the data storage unit information 204 indicating the same connection management ID as the acquired connection management ID is extracted from the stored data storage unit information 204 (FIG. 4).
  • the cooperative memory management unit 18 uses the individual data storage unit ID in the individual data storage information 206 (FIG. 4) associated with the extracted data storage unit information 204 and the individual exclusive control information 205 (FIG. 4).
  • the individual exclusive control ID is acquired.
  • the cooperative memory management unit 18 maps the individual data storage unit 21 using the acquired individual data storage unit ID as a key. More specifically, when implemented in C language, the operation is to map the shared memory by the map file. Further, the cooperative memory management unit 18 generates a semaphore for exclusive control using the acquired individual exclusive control ID as a key.
  • the linkage memory management unit 18 generates an open handle, extracts the process management table 203 (FIG. 4) indicating the same initialization handle as the initialization handle notified from the conversion unit 12,
  • the generated open handle information is stored in the open handle information 208 associated with the extracted process management table 203, the semaphore key is stored in the individual exclusive control information column, and the shared memory is stored in the individual data storage information column. (The address of the individual data storage unit) is stored, the output edge ID notified from the conversion unit 12 is stored in the input / output edge column, and the open handle is returned to the conversion unit 12.
  • the number of blocks and the block size in each individual data storage unit can be calculated from the database design information and the past update data arrival log.
  • the memory area secured by the data storage unit is optimized, the occurrence of memory overflow during execution of data linkage processing is suppressed, and the securing of unused memory that is not used is avoided.
  • each CDW processing unit 13 can update two or more records in the cooperation destination table in one transaction.
  • the receiving unit 11 extracts the process management table 203 (FIG. 4) of the cooperation management information 17 in which the open handle that matches the open handle acquired in the open process is extracted, and is associated with the extracted process management table 203.
  • the semaphore of the individual exclusive control unit 20 is acquired from the individual exclusion control information of the open handle information 208, the right to write to the individual data storage unit 21 is acquired, and the updated record received from the CDC 22 is stored in the individual data storage unit 21. Write.
  • the semaphore of the individual exclusive control unit 20 is released, and a message for interprocess communication is written in the data processing control unit 16.
  • the data processing control unit 16 has a FIFO (First-In First-Out) structure having a list structure as shown in FIG.
  • the receiving unit 11 writes a message for inter-process communication in the data processing control unit 16 and then releases the semaphore of the read control flag. Thereby, the data writing event is notified to the conversion unit 12, and the update order restriction of the cooperation source database 3 can be observed and the update data can be reflected in the cooperation destination database 5.
  • FIG. 9 shows processing of one transaction in the conversion unit 12.
  • the conversion unit 12 acquires the initialization handle from the process management table 203 in which the process ID of itself (the conversion unit 12) is described (FIG. 4), and stores the initialization handle in the individual data storage unit 21 linked to the initialization handle.
  • a connection destination address is acquired (S100).
  • the updated record is read from the individual data storage unit 21, and the read updated record is converted to generate the update reflected data.
  • the conversion unit 12 acquires an open handle by referring to the link management information 17 using the output edge acquired from the link connection information 24 (FIG. 5) as a key when performing the conversion process.
  • the conversion unit 12 acquires the semaphore of the individual exclusive control unit 20 managed using the open handle as a key (S101).
  • the conversion unit 12 When all the blocks of the individual data storage unit 21 of the CDW processing unit 13 are in an unprocessed state, that is, when the buffer of the individual data storage unit 21 is filled with data that has not been dequeued by the CDW processing unit 13, the conversion unit 12 In this embodiment, the transaction arrival frequency per unit time is accumulated in the update data arrival log 15 and the statistical processing result is stored in the format shown in FIG. Since the cooperative memory management unit 18 calculates the number of blocks of the individual data storage unit based on the information of FIG. 6, all the blocks of the individual data storage unit are unprocessed when the CDW processing unit 13 is operating normally. The occurrence of an event can be avoided.
  • the conversion unit 12 when the conversion unit 12 writes update reflected data to a plurality of CDW processing units 13, it is necessary to acquire all the open handles corresponding to each output edge and the semaphores of the individual exclusive control unit 20 ( S102).
  • the conversion unit 12 performs the writing process to the individual data storage unit 21 that has acquired the open handle and the semaphore in parallel (S103).
  • the conversion unit 12 releases the semaphore of each individual exclusive control unit 20 corresponding to the output edge (S104).
  • the conversion unit 12 confirms that all the semaphores of the individual exclusive control unit 20 have been released (S105).
  • the conversion unit 12 acquires the semaphore of the data processing control unit 16 of the CDW processing unit 13 (S106).
  • the conversion unit 12 performs an enqueue process to the data processing control unit 16, and stores a set of the output edge and update record information of the update data information 310 shown in FIG. 8 for each output edge (S107).
  • the conversion unit 12 releases the semaphore of the data processing control unit 16 of the CDW processing unit 13 (S108).
  • the processing of the conversion unit 12 for one transaction data is completed (S109).
  • the CDW processing unit 13 extracts the process management table 203 (FIG. 4) in which the same process ID as the process ID of itself (CDW processing unit 13) is shown, and the initialization stored in the extracted process management table 203 A handle is acquired (S401). Next, the CDW processing unit 13 acquires the individual exclusive control information based on the data storage unit ID of the data storage unit information associated with the extracted process management table 203, and the individual data read by the CDW processing process The storage unit ID is acquired (S402). Next, the write-waiting data stored in the data processing control unit 16 (FIFO structure) that controls data writing to the CDW processing unit 13 is dequeued.
  • the data processing control unit 16 FIFO structure
  • the data processing control unit 16 stores a write request message corresponding to the update reflected data from the conversion unit 12 accumulated in the individual data storage unit 21. At this time, a plurality of write request messages may be collectively dequeued from the data processing control unit 16 (S403).
  • the CDW processing unit 13 locks each individual data storage unit 21 based on the previously acquired individual exclusive control information and copies the stored update reflection data to the local memory (S404). After obtaining all the write request messages from the data processing control unit 16 and copying all the update reflected data to the local memory (S405), the CDW processing unit 13 dequeues the write request message from the data processing control unit 16.
  • an SQL statement to be issued to the DBMS (DataBase Management System) of the cooperation destination database 5 is generated (S406).
  • the CDW processing unit 13 sequentially executes the SQL statements generated in S406 (S407).
  • the commit process is not performed for each SQL1 sentence.
  • the CDW processing unit 13 issues a commit process to the DBMS of the cooperation destination database 5.
  • the number of commit processes issued by the CDW processing unit 13 is reduced, and an effect of reducing the communication processing time with the cooperation destination database is obtained (S409).
  • the database linkage apparatus distributes update data to a plurality of linkage destination databases 5 for a plurality of linkage source databases 3.
  • the CDW process part which receives the input of conversion data from the some conversion part 12 which converts the specification of update data, and the some conversion part 12, and reflects in the cooperation destination database 5 13 is provided with a data linkage interlocking buffer management unit 14.
  • a data linkage interlocking buffer management unit 14 for each CDW processing unit 13, an individual data storage unit 21 and an individual exclusive control unit 20 for each conversion unit 12 are configured, so that data writing from the conversion unit 12 is performed in parallel. Processing is possible and throughput is improved.
  • Integrated linkage information with data linkage definition, data linkage processing unit, data linkage linked buffer management unit includes at least a conversion unit and a CDW processing unit
  • the data linkage interlocking buffer management unit includes a data processing control unit that controls data transfer between the conversion unit and the CDW processing unit for each CDW processing unit, and a data storage unit configured for each CDW processing unit
  • the data storage unit includes an individual exclusive control unit and an individual data storage unit for each conversion unit that writes data to the CDW processing unit, Database linkage device that asynchronously processes the acquisition and conversion of update records and the reflection to the linkage destination database table when the linkage source database table is updated, and reflects multiple update records to the linkage destination database in a batch Explained.
  • the data linkage interlocking buffer management unit has linkage management information
  • the linkage management information includes the correspondence relationship between the individual data storage unit configured by the process unit that performs the conversion process of the linkage source database for each CDW processing unit that performs the data reflection process, and the individual exclusive control information, It has been described that the individual data storage unit is provided with a plurality of connection management IDs and the total column length to be linked.
  • the integrated linkage information includes information that configures the linkage information to the linkage destination database in units of columns of the linkage source database as a combination of the linkage source database column identifier and the linkage destination database column identifier,
  • the conversion unit of the data cooperation processing unit writes data to the CDW processing unit using the information
  • the data is collectively written in all the individual data storage units that are the data write destinations, and the data processing control is performed after the writing is completed. It has been explained that writing is notified to a plurality of the CDW processing units via the unit, and the plurality of CDW processing units are operated in parallel.
  • An individual data storage unit for a specific CDW processing unit to read data is composed of a plurality of data buffers, and by providing individual exclusive control for each individual data storage unit, parallel writing from a plurality of conversion units is possible. Explained what to do.
  • the linked memory management unit includes statistical processing means for the log
  • the block size in the individual data storage unit is calculated from the product of the total column length defined in the linkage information targeted by a specific CDW processing unit when the linkage information is updated and the number of records per unit arrival, and the number of data arrivals and blocks Calculate the data buffer size per linkage source table from the product of the number of linkage definitions for the same linkage source table and the size, and calculate the data buffer size for all linkage source tables for the specific CDW processing unit
  • the calculation of the capacity of the individual data storage unit has been described.
  • Embodiment 2 An example of adding a new database definition between the cooperation source database 3 and the cooperation destination database 5 will be described with reference to the drawings.
  • the CDW processing unit 13 notifies the linked memory management unit 18 of the added output edge ID and its own CDWid. At this time, it is assumed that the connection output information 211 (FIG. 5) including the added output edge ID has been generated.
  • the cooperative memory management unit 18 extracts the connection output information 211 indicating the notified output edge ID and CDWid, and acquires the connection management ID indicated in the extracted connection output information 211 (S301).
  • the cooperation memory management unit 18 extracts the process management table 203 (FIG. 4) of the cooperation management information 17 indicating the same process ID as the notified CDWid, and associates it with the extracted process management table 203.
  • the data storage unit information 204 showing the same connection management ID as the connection management ID acquired in S301 is searched (S302). If the corresponding data storage unit information 204 cannot be extracted, that is, if the connection management ID acquired in S301 is not registered (YES in S303), a new individual data storage unit 21 and individual exclusive control unit 20 are generated. Since it is necessary, the cooperative memory management unit 18 acquires additional information (record length) at the time of execution from the connection output information 211 extracted in FIG. .
  • the linked memory management unit 18 refers to the update data arrival log 15 and selects a record having a matching connection management ID acquired from the connection output information 211, and calculates the average number of transactions / unit time (Stn) and the average number of records. / The value of transaction (St1) is acquired (S304).
  • the conversion unit 12 to be a cooperation source is added, that is, when a table of the cooperation source database is newly added, the average number of transactions and the average number of records are not recorded in the update data arrival log There is.
  • a predetermined value specified in advance is assigned as the average number of transactions and the average number of records, and is used as an index for calculating the size of the individual data storage unit 21.
  • the cooperative memory management unit 18 acquires a buffer having a size corresponding to (Stn ⁇ number of duplicate connection management IDs) ⁇ St1 ⁇ S1 as an individual data storage unit, and stores the individual data in the individual data storage information 206 (FIG. 4).
  • a connector for the buffer is added to the storage unit ID column, and a line for storing the connection management ID is added to the connection management ID column (S305).
  • the CDW processing unit 13 establishes access to the individual data storage unit 21 (S306).
  • connection management ID if there is a connection management ID in S303, the runtime additional information (record length) is acquired from the connection output information 211 acquired in S301, and the acquired runtime additional information (record length) is used for the search in S302. The result is added to the total column length (Sl) to be linked in the individual data storage information 206 associated with the data storage unit information 204 (FIG. 4) obtained (S307). Further, the cooperative memory management unit 18 adds the connection management ID duplication number of the individual data storage information 206 (S308).
  • the linked memory management unit 18 acquires the latest average transaction count / unit time (Stn) and average record count / transaction (Stl) values from the update data arrival log 15 using the connection management ID as a key, and Stn * St1 * S1 is calculated, and the incremental capacity of the individual data storage unit is calculated (S309).
  • the cooperative memory management unit 18 searches the conversion unit connection information (FIG. 5) for the process identifier of the conversion unit 12 based on the output edge ID, and acquires the process identifier and the addition target CDW process.
  • the execution of the access blocking process to the individual data storage unit 21 is notified to the unit 13 (S310).
  • the cooperative memory management unit 18 expands the capacity of the individual data storage unit 21 (S311).
  • the cooperative memory management unit 18 notifies the conversion unit 12 and the CDW processing unit 13 that have been blocked from accessing the individual data storage unit 21 that the reopening process has been performed (S312).
  • the conversion unit 12 notifies the association memory management unit 18 of the initialization handle and the output edge ID.
  • the cooperative memory management unit 18 acquires the CDWid and the connection management ID from the connection destination information 211 (FIG. 5) of the cooperative connection information using the output edge ID as a key.
  • the cooperation memory management unit 18 extracts the process management table 203 (FIG. 4) of the cooperation management information 17 in which the same process ID as the acquired CDWid is shown, and is associated with the extracted process management table 203.
  • Data storage unit information 204 having the same connection management ID as the acquired connection management ID is extracted.
  • the cooperative memory management unit 18 acquires the individual data storage unit ID of the individual data storage information associated with the extracted data storage unit information 204 and the individual exclusive control ID of the individual exclusive control information.
  • the cooperative memory management unit 18 maps the individual data storage unit 21 using the acquired individual data storage unit ID as a key. More specifically, when implemented in C language, the operation is to map the shared memory by the map file. Furthermore, the cooperative memory management unit 18 generates a semaphore for exclusive control using the individual exclusive control ID as a key.
  • the linkage memory management unit 18 generates an open handle, extracts the process management table 203 (FIG. 4) indicating the same initialization handle as the initialization handle notified from the conversion unit 12, The generated open handle information is stored in the open handle information 208 associated with the extracted process management table 203, the semaphore key is stored in the individual exclusive control information column, and the shared memory is stored in the individual data storage information column. (The address of the individual data storage unit) is stored, the output edge ID notified from the conversion unit 12 is stored in the input / output edge column, and the open handle is returned to the conversion unit 12.
  • the processing of the conversion unit 12 and the CDW processing unit 13 related to the updated cooperative connection information 24 is interrupted, and the data storage unit 19 is opened. Processing will occur.
  • the processing of all the conversion units and the CDW processing unit sharing the buffer has been interrupted.
  • the generation target of the open processing is limited to only the conversion unit 12 corresponding to the table added as the cooperation source table and the CDW processing unit 13 operating corresponding to the database constituting the cooperation destination table. Thus, the throughput of the database linkage apparatus can be improved.
  • the CDW processing unit 13 other than the CDW processing unit 13 and the CDW processing unit 13 are updated.
  • Generation of a new individual data storage unit 21 corresponding to the CDW processing unit 13 and processing of the CDW processing unit 13 are performed without affecting the operation of the conversion unit 12 other than the conversion unit 12 that is the transfer destination of the reflected data.
  • the capacity of the existing individual data storage unit 21 can be changed.
  • Linked memory that refers to the link destination table information when the integrated link information is updated, and calculates the data length written from the conversion unit based on the column configuration information of the new link destination table added to the link destination table information
  • a database cooperation apparatus has been described in which the capacity of an individual data storage unit is calculated according to the data length of a record newly added as a cooperation destination database, and the individual data storage unit can be expanded.
  • Embodiment 3 FIG.
  • a secondary storage storage unit 26 for temporarily storing data in the individual data storage unit 21 is provided in addition to hardware for mounting the database linkage apparatus 1 will be described with reference to the drawings. To do.
  • FIG. 12 shows a system configuration example according to the present embodiment.
  • the data linkage design unit 6 and the data linkage design information 7 are not shown for reasons of drawing. However, as in FIG. Assume that the data linkage design information 7 is included.
  • a data saving device 27 including a secondary storage unit 26 that is a secondary storage device is provided in addition to the configuration of the database cooperation device 1 shown in FIG. 1, a data saving device 27 including a secondary storage unit 26 that is a secondary storage device is provided.
  • the data saving device 27 and the database cooperation device 1 may be mounted on the same hardware device or different hardware devices.
  • the structure which utilizes the data evacuation apparatus 27 with the some database cooperation apparatus 1 may be sufficient.
  • the individual data storage unit 21 is generated on the main storage device (shared memory).
  • the CDW processing unit 13 cannot erase the update reflected data stored in the individual data storage unit 21 because the CDW processing unit 13 cannot perform the writing process to the cooperation destination database 5.
  • the memory in which the individual data storage unit 21 is mounted is depleted and the processing of the conversion unit 12 is in a waiting state, so that the influence is propagated to the reception unit 11 and reception of update data from the CDC 22 is delayed. End up.
  • the update reflected data to be stored in the individual data storage unit 21 is temporarily saved in the secondary storage unit 26.
  • the processing unit 12 is prevented from waiting for processing, and the update data arrival delay from the CDC 22 is prevented from occurring. That is, when the accumulated amount of the update reflected data in the individual data storage unit 21 exceeds a predetermined level and the overflow is close, the cooperative memory management unit 18 stores the individual data storage unit 21 for update reflected data in the secondary storage storage unit 26. Is generated.
  • the conversion unit 12 sends the update reflected data to the individual data storage unit 21 in the main storage device (shared memory) before the individual data storage unit 21 is generated in the secondary storage unit 26 by the cooperative memory management unit 18.
  • the update reflected data is written in the individual data storage unit 21 in the secondary storage unit 26.
  • the linked memory management unit 18 stores the individual data for the updated records in the secondary storage unit 26.
  • the storage unit 21 is generated.
  • the receiving unit 11 stores the updated record in the individual data storage unit 21 in the main storage device (shared memory) before the individual data storage unit 21 is generated in the secondary storage unit 26 by the cooperative memory management unit 18.
  • the updated record is written in the individual data storage unit 21 in the secondary storage unit 26.
  • Embodiment 4 FIG. In the present embodiment, an example in which the secondary storage control unit 28 is provided in the data saving device 27 in addition to the configuration of the database cooperation device 1 illustrated in FIG. 12 will be described.
  • FIG. 13 shows a system configuration example according to the present embodiment.
  • the data linkage design unit 6 and the data linkage design information 7 are omitted for reasons of drawing.
  • the database linkage apparatus 1 includes the data linkage design unit 6. It is assumed that data linkage design information 7 is included.
  • a secondary storage control unit 28 is added to the data saving device 27 compared to the configuration of FIG. 12.
  • the write confirmation process is performed when data (updated record, update reflected data) is written in the secondary storage unit 26, so the writing speed is slow, and the database linkage apparatus 1 Throughput may be significantly reduced.
  • the secondary storage control unit 28 associates the identifier of the data cooperation processing unit 23 that is the write source, the secondary storage storage unit 26 that is the write destination, and the identifier of the data cooperation processing unit 23 that is the read target. Hold.
  • the secondary storage control unit 28 requests the secondary storage storage unit 26 to write to the secondary storage storage unit 26
  • the secondary storage control unit 28 receives the identifier of the requesting data linkage processing unit 23, and the corresponding save area.
  • the secondary storage control unit 28 acquires the identifier of the data cooperation processing unit 23 that is the request source, and the corresponding secondary storage storage unit 26 is performed, and only the confirmation process is performed by the reading process from the secondary storage unit 26. More specifically, when a relational database is used as the data saving device 27, the commit is performed when the data read request arrives at the secondary storage control unit 28 without committing at the time of data writing, and the confirmed data is read. It is configured to return to the request source.
  • the secondary storage control unit 28 writes the updated record / update reflected data to the individual data storage unit 21 in the secondary storage storage unit 26 by the reception unit 11 / conversion unit 12.
  • the receiving unit 11 does not confirm the writing, and when the conversion unit 12 / CDW processing unit 13 reads the updated record / update reflected data from the individual data storage unit 21 in the secondary storage unit 26. / Confirm writing of updated record / update reflected data by the conversion unit 12. For this reason, it is possible to suppress a decrease in throughput of the database linkage apparatus 1 due to repeated occurrence of the write confirmation process.
  • the database cooperation system provided with the secondary storage storage part which stores data as a secondary storage, and the secondary storage control part was demonstrated. More specifically, Secondary storage controller Control the writing process and reading process to the secondary storage unit, Do not execute the data write confirmation process during the write process, perform the data confirm process during the read process, It has been explained that the processing time of the database linkage apparatus with respect to the saving apparatus can be increased by reducing the number of execution times of the data confirmation process.
  • FIG. 15 is a diagram illustrating an example of hardware resources of the database linkage apparatus 1 described in the first to fourth embodiments.
  • the configuration in FIG. 15 is merely an example of the hardware configuration of the database linkage device 1, and the hardware configuration of the database linkage device 1 is not limited to the configuration described in FIG. Also good.
  • the database linkage apparatus 1 includes a CPU 911 (also referred to as a central processing unit, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, and a processor) that executes a program.
  • the CPU 911 is connected to, for example, a ROM (Read Only Memory) 913, a RAM (Random Access Memory) 914, a communication board 915, a display device 901, a keyboard 902, a mouse 903, and a magnetic disk device 920 via a bus 912. Control hardware devices.
  • the CPU 911 may be connected to an FDD 904 (Flexible Disk Drive), a compact disk device 905 (CDD), a printer device 906, and a scanner device 907.
  • FDD 904 Flexible Disk Drive
  • CDD compact disk device
  • printer device 906 printer device 907
  • a storage device such as an SSD (Solid State Drive), an optical disk device, or a memory card (registered trademark) read / write device may be used.
  • the RAM 914 is an example of a volatile memory.
  • the storage media of the ROM 913, the FDD 904, the CDD 905, and the magnetic disk device 920 are an example of a nonvolatile memory. These are examples of the storage device.
  • the “individual data storage unit 21” described in the first to fourth embodiments is configured on the RAM 914 as a shared memory.
  • a communication board 915, a keyboard 902, a mouse 903, a scanner device 907, an FDD 904, and the like are examples of input devices.
  • the communication board 915, the display device 901, the printer device 906, and the like are examples of output devices.
  • the communication board 915 is connected to the network as shown in FIG.
  • the communication board 915 may be connected to a LAN (local area network), the Internet, a WAN (wide area network), a SAN (storage area network), or the like.
  • the magnetic disk device 920 stores an operating system 921 (OS), a window system 922, a program group 923, and a file group 924.
  • the programs in the program group 923 are executed by the CPU 911 using the operating system 921 and the window system 922.
  • the RAM 914 temporarily stores at least part of the operating system 921 program and application programs to be executed by the CPU 911.
  • the RAM 914 stores various data necessary for processing by the CPU 911.
  • the ROM 913 stores a BIOS (Basic Input Output System) program
  • the magnetic disk device 920 stores a boot program.
  • BIOS Basic Input Output System
  • the BIOS program in the ROM 913 and the boot program in the magnetic disk device 920 are executed, and the operating system 921 is activated by the BIOS program and the boot program.
  • the program group 923 stores programs for executing the functions described as “ ⁇ unit” (except for “individual data storage unit 21”, the same applies hereinafter) and “ ⁇ means” in the description of the first to fourth embodiments. Has been.
  • the program is read and executed by the CPU 911.
  • Information, data, signal values, variable values, and parameters stored in a storage medium such as a disk or memory are read out to the main memory or cache memory by the CPU 911 via a read / write circuit.
  • the read information, data, signal value, variable value, and parameter are used for CPU operations such as extraction, search, reference, comparison, calculation, calculation, processing, editing, output, printing, and display.
  • Information, data, signal values, variable values, and parameters are stored in the main memory, registers, cache memory, and buffers during the CPU operations of extraction, search, reference, comparison, calculation, processing, editing, output, printing, and display. It is temporarily stored in a memory or the like.
  • the arrows in the flowcharts described in Embodiments 1 to 4 mainly indicate input / output of data and signals.
  • Data and signal values are recorded on a recording medium such as a memory of the RAM 914, a flexible disk of the FDD 904, a compact disk of the CDD 905, a magnetic disk of the magnetic disk device 920, other optical disks, a mini disk, and a DVD.
  • Data and signals are transmitted online via a bus 912, signal lines, cables, or other transmission media.
  • Embodiments 1 to 4 what is described as “to part” and “to means” may be “to circuit”, “to device”, and “to device”. It may be “ ⁇ step”, “ ⁇ procedure”, or “ ⁇ process”. That is, the data processing apparatus according to the present invention can be grasped as a method by the steps, procedures, and processes shown in the flowcharts described in the first to fourth embodiments. Further, what is described as “ ⁇ unit” and “ ⁇ means” may be realized by firmware stored in the ROM 913. Alternatively, it may be implemented only by software, or only by hardware such as elements, devices, substrates, and wirings, by a combination of software and hardware, or by a combination of firmware.
  • Firmware and software are stored as programs in a recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, and a DVD.
  • the program is read by the CPU 911 and executed by the CPU 911.
  • the program causes the computer to function as “to part” and “to means” in the first to fourth embodiments.
  • the computer executes the procedures and methods of “to part” and “to means” in the first to fourth embodiments.
  • the database cooperation device 1 shown in the first to fourth embodiments includes a CPU as a processing device, a memory as a storage device, a magnetic disk, a keyboard as an input device, a mouse, a communication board, a display device as an output device, and a communication device.
  • a computer including a board or the like.
  • the functions indicated as “ ⁇ unit” and “ ⁇ means” are realized by using these processing devices, storage devices, input devices, and output devices.
  • 1 database linkage device 2 linkage source system, 3 linkage source database, 4 linkage destination system, 5 linkage destination database, 6 data linkage design department, 7 data linkage design information, 8 integration linkage information, 9 linkage source table information, 10 linkage Destination table information, 11 receiving unit, 12 converting unit, 13 CDW processing unit, 14 data linkage interlocking buffer management unit, 15 update data arrival log, 16 data processing control unit, 17 linkage management information, 18 linkage memory management unit, 19 Data storage unit, 20 individual exclusive control unit, 21 individual data storage unit, 22 CDC, 23 data linkage processing unit, 24 linkage connection information.

Abstract

La présente invention concerne un dispositif de traitement de données comportant une unité de gestion de mémoire en coopération (18) qui génère une unité de stockage de données individuelle (21) pour l'écriture d'un enregistrement mis à jour à partir d'une capture de changements aux données CDC (22), et une unité de stockage de données individuelle (21) pour l'écriture de données correspondant à la mise à jour générées par une unité de conversion (12) à partir de l'enregistrement mis à jour. L'unité de conversion (12) effectue une lecture de l'enregistrement mis à jour provenant de l'unité de stockage de données individuelle (21) de manière asynchrone avec une unité de réception (11), génère des données correspondant à la mise à jour, et l'écriture des données correspondant à la mise à jour dans l'unité de stockage de données individuelle (21). Une unité de traitement d'entrepôt de données central CDW (13) effectue une lecture des données correspondant à la mise à jour provenant de l'unité de stockage de données individuelle (21) de manière asynchrone avec l'unité de réception (11) et l'unité de conversion (12) et la mise à jour d'une table de destinations en coopération.
PCT/JP2010/073608 2010-12-27 2010-12-27 Dispositif de traitement de données WO2012090281A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2012550608A JPWO2012090281A1 (ja) 2010-12-27 2010-12-27 データ処理装置
PCT/JP2010/073608 WO2012090281A1 (fr) 2010-12-27 2010-12-27 Dispositif de traitement de données

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2010/073608 WO2012090281A1 (fr) 2010-12-27 2010-12-27 Dispositif de traitement de données

Publications (1)

Publication Number Publication Date
WO2012090281A1 true WO2012090281A1 (fr) 2012-07-05

Family

ID=46382431

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2010/073608 WO2012090281A1 (fr) 2010-12-27 2010-12-27 Dispositif de traitement de données

Country Status (2)

Country Link
JP (1) JPWO2012090281A1 (fr)
WO (1) WO2012090281A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014106568A (ja) * 2012-11-22 2014-06-09 Mitsubishi Electric Corp 表示操作器及び画面パターン情報更新システム
JP7457282B2 (ja) 2020-02-28 2024-03-28 京セラドキュメントソリューションズ株式会社 データ連携システムおよび構成変更システム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH05324426A (ja) * 1992-05-22 1993-12-07 Toshiba Corp フォーマット変換装置
JPH0883221A (ja) * 1994-09-14 1996-03-26 Toshiba Corp データ変換装置
JPH10340217A (ja) * 1997-05-09 1998-12-22 At & T Corp データベースネットワーク及びデータ更新複製方法
JP2005322159A (ja) * 2004-05-11 2005-11-17 Canon Inc データ連携システム及び方法
JP2005333434A (ja) * 2004-05-20 2005-12-02 Matsushita Electric Ind Co Ltd 無線モジュール

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH05324426A (ja) * 1992-05-22 1993-12-07 Toshiba Corp フォーマット変換装置
JPH0883221A (ja) * 1994-09-14 1996-03-26 Toshiba Corp データ変換装置
JPH10340217A (ja) * 1997-05-09 1998-12-22 At & T Corp データベースネットワーク及びデータ更新複製方法
JP2005322159A (ja) * 2004-05-11 2005-11-17 Canon Inc データ連携システム及び方法
JP2005333434A (ja) * 2004-05-20 2005-12-02 Matsushita Electric Ind Co Ltd 無線モジュール

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014106568A (ja) * 2012-11-22 2014-06-09 Mitsubishi Electric Corp 表示操作器及び画面パターン情報更新システム
JP7457282B2 (ja) 2020-02-28 2024-03-28 京セラドキュメントソリューションズ株式会社 データ連携システムおよび構成変更システム

Also Published As

Publication number Publication date
JPWO2012090281A1 (ja) 2014-06-05

Similar Documents

Publication Publication Date Title
US11003689B2 (en) Distributed database transaction protocol
US11036540B2 (en) Transaction commit operations with thread decoupling and grouping of I/O requests
US20220067025A1 (en) Ordering transaction requests in a distributed database according to an independently assigned sequence
US9959178B2 (en) Transactional and parallel log replay for asynchronous table replication
US9230002B2 (en) High performant information sharing and replication for single-publisher and multiple-subscriber configuration
JP4384633B2 (ja) 非同期情報共有システム
US9934263B1 (en) Big-fast data connector between in-memory database system and data warehouse system
US11263236B2 (en) Real-time cross-system database replication for hybrid-cloud elastic scaling and high-performance data virtualization
US20160246864A1 (en) Relaxing transaction serializability with statement-based data replication
CN113826084A (zh) 流媒体数据的高效进程外重组
CN101375241A (zh) 集群文件系统中的有效数据管理
AU2006333375B2 (en) Method and mechanism for loading XML documents into memory
Chohan et al. Database-agnostic transaction support for cloud infrastructures
JP4432087B2 (ja) データベース更新管理システム、プログラムおよび方法
JP5331050B2 (ja) データ同期システム、データ同期方法、情報処理装置、情報処理方法、およびプログラム
WO2012090281A1 (fr) Dispositif de traitement de données
JPWO2008105099A1 (ja) アプリケーション連携制御プログラム、アプリケーション連携制御方法およびアプリケーション連携制御装置
CN111459882A (zh) 分布式文件系统的命名空间事务处理方法和装置
US20110191549A1 (en) Data Array Manipulation
JP5494915B2 (ja) レプリケーションシステム、マスタサーバ、レプリカサーバ、レプリケーション方法、及びプログラム
US11921691B2 (en) Low latency demultiplexer for propagating ordered data to multiple sinks
Lev-Ari et al. Quick: a queuing system in cloudkit
JP2008146420A (ja) Ｉ／ｏ要求の処理順序を最適化するためのコンピュータプログラム
Diaz et al. Working with NoSQL Alternatives
Arora et al. Oracle® Streams for Near Real Time Asynchronous Replication

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 10861340

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2012550608

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 10861340

Country of ref document: EP

Kind code of ref document: A1