JP3797143B2 - Bulk loading system, bulk loading method, and bulk loading program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a bulk loading system, a bulk loading method, and a bulk loading program, and more particularly, bulk loading that enables the identification information and attribute information of entries described in an input file in a directory service to be collectively loaded into a directory server. The present invention relates to a system, a bulk loading method, and a bulk loading program.
[0002]
[Prior art]
Conventionally, in directory services such as personal information management, network management, and file management, resource information such as people and computers is represented by entries, and each entry uses a plurality of accompanying information such as names as attribute information (attributes). keeping.
Each entry is generally classified into a hierarchical structure according to the organization or region to which the entry belongs, and is centrally managed using a directory server.
[0003]
As an example of a technique using such a directory server, for example, in Japanese Patent Application Laid-Open No. 2000-242538, a directory search system capable of improving the search speed and reducing the storage capacity by using the ancestor relationship of entries. Technology has been proposed.
When registering entry and attribute data in the storage device, the directory server used in this directory search system generates an index (an ancestor relationship table) that is used for speeding up the directory search. Register together.
[0004]
In other words, the directory server normally manages entry data, attribute data, and index data as directory data, and various indexing methods have been developed. Using this index data, directory search can be performed at high speed. It is configured as described above.
[0005]
In addition, the technologies related to the directory service include various technologies for the purpose of speeding up search processing and improving memory efficiency so that a large-scale directory can be handled more quickly and efficiently. Is disclosed.
[0006]
By the way, as a general directory data generation method, a utility is used to generate directory data for each entry, and all entries are described in a file according to an arbitrary format. There is a method of using a tool that reads and loads directory data in a lump. The latter can load directory data in a lump, and thus can deal with large-scale directory data quickly and efficiently.
[0007]
For example, in LDAP (Lightweight Directory Access Protocol), which is a standard directory service on the Internet, a file format for data exchange called LDIF (LDAPPDataInterchangeFormat) is defined in RFC2849. Many of them have the ability to read and write all files in a batch.
[0008]
In addition, in order to load LDIF files and load directory data all at once, two methods are usually adopted, one is a method of loading data according to the LDAP protocol, and the other is This is a method of loading data as a database application when a database is used for the back end.
However, in both methods, the data described in the input file is loaded into the directory server for each entry to generate the directory data.
[0009]
(First conventional example)
As the former method, for example, Japanese Patent Application Laid-Open No. 11-345234 has proposed a technique of a data processing apparatus that reads data collectively from a file according to a common format and performs arbitrary processing.
This data processing apparatus can register candidate employees in advance by using a file in a format called CSV (Comma Separated Value), and can read the data of these employees all at once.
[0010]
(Second conventional example)
Similarly, as the former method, for example, Japanese Patent Application Laid-Open No. 10-326285 proposes a document management system technique for collectively registering documents using the same CSV file.
This document management system includes a database system storing attributes and file names of all documents, a management data file for creating data required for document management in CSV format by reading data from the database system, and the created management A batch registration file in CSV format created by reading and editing data of a data file, a batch registration interface for reading a batch registration file and registering a document, an object-oriented database system in which the batch registration interface actually registers, It consists of an error log file that is output when an error occurs during registration processing.
[0011]
The batch registration process of the document management system having the above configuration operates as follows.
First, the first line of the batch registration file is read and analyzed, and the registration items are associated with the field order.
Subsequently, the second and subsequent lines of the batch registration file are read line by line, and a registration data array is generated based on the association obtained in the preprocessing.
If there is no error, registration is performed in the database using the generated registration data array.
If there is an error, the error content and error record are added to the log file in CSV format.
[0012]
(Third conventional example)
On the other hand, as the latter method, there is a bulk loading method as a method of loading data stored in a database all at once from a file.
As an example of applying the bulk loading method to an object-oriented database, BulkLoading into OODB published in 1994, Proceedings of VLD Conference, pages 120-131 (ProceedingsofVLDBConference, 1994, Pages 120-131). : JanetL., Entitled APerformanceStudy. There is a paper by Wiener et al.
[0013]
Also, as an example of further expanding the bulk loading method described in this paper, 1995, Proceedings of VLD BD Conference, p. 30-41 (ProceedingsofVLDBConference, 1995, Pages 30-41) Published in JanetL., Entitled OODBBulkLoadingRevised: The Partitioned-ListApproach. There is a paper by Wiener et al.
[0014]
The bulk loading method described in these papers is basically the following method.
Newly read data (objects) are prepared in advance as text files, and each object is assigned a unique number. A text file in which objects that have already been read are described as IDs for all objects. (Identification code) is assigned and managed by an ID map using this ID.
[0015]
In other words, each object that has already been read is managed in the ID map, and an object to be newly read (object to be referenced) is a ToDo list to which an ID map and a temporary ID are assigned (when there is a reverse reference, the InvToDo list) ).
[0016]
In addition, for an object to be newly read, an Update list including information that needs to be updated is generated using an ID map and a ToDo list (or InvTodo list).
Then, the text file of the newly read object is read again, and an object including the newly read object is generated while referring to the Update list.
[0017]
[Problems to be solved by the invention]
However, the above-described prior art is not limited to the case where the utility provided by the directory service is used (the former method) but also the case where the directory data is collectively loaded using the input file (the latter method). There is a problem that data must be loaded for each entry and the processing speed cannot be improved.
[0019]
  In addition, there is a problem that the directory data is not efficiently arranged so that the search performance of the directory service after the data is loaded is improved.
[0020]
The present invention has been made to solve the above problem, and is a bulk load system capable of clustering and collectively loading the directory data in the directory service at high speed and improving the search performance of the directory service. The object is to provide a bulk loading method and a bulk loading program.
[0021]
[Means for Solving the Problems]
  In order to achieve this object, the bulk load system according to claim 1 of the present invention comprises a data processing device controlled by a bulk load control means, an input file analysis means, and a hash table management means. , A reference list management means, and a data page generation means, wherein the input file analysis means analyzes the input file and generates data for each entry.Read the first time to read,The hash table management means manages the identification information of the entry in a hash table, and the reference list management meansA reference list is generated based on a temporary identification code map and / or a temporary reference list having a temporary identification code for the entry of the input file, which is different from the actual identification code for the entry read by the input file analyzing means. The reference relation of the entry is managed by the reference list, the bulk load control unit generates an update list based on the reference list, and the input file analysis unit converts the data of the entry of the input file On the other hand, the second reading is performed, and the data page generation unit generates the input file entry data acquired in the second reading in the database storage format based on the update list.As a configuration.
[0022]
  In this way, the reference list management means manages the reference relationship of the entries in the reference list, that is, acquires the reference relationship of the entries in advance, so that the entry data included in the input file can be loaded efficiently and at high speed. .Further, the entry reference relationship can be stored in the temporary identification code map and / or the temporary reference list every time one entry is read. Furthermore, even when the input file is composed of a plurality of entries, data of a plurality of entries can be loaded in a batch.
[0023]
  The invention according to claim 2 is the bulk load system according to claim 1, wherein the reference list includes an actual entry for the entry read by the input file analysis means.identificationA code, a parent identification code for the entry representing a superordinate concept, and a child identification code for the entry representing a subordinate concept.
[0024]
In this way, the entry reference relationship can be expressed by a parent-child relationship based on the entry data, and the reference list can be easily used.
[0025]
According to a third aspect of the present invention, in the bulk load system according to the second aspect, the reference list has the following identification code indicating a sibling relationship between the entries with respect to the entries at the peer level: As a configuration.
[0026]
In this way, the entry reference relationship can be represented by a sibling relationship based on the entry data, and the reference list can be easily used.
[0031]
  Claims4The invention described is the above claim.In any one of 1-3In the described bulk loading system, the input file analyzing means reads the first time and the second time of the data of all entries of the input file.
[0032]
In this way, even if the input file consists of a plurality of entries, the data of all the entries in the input file can be loaded in a batch.
[0033]
  Further, the claims of the present invention5In the described bulk load system, a data processing apparatus includes a bulk load control unit, an input file analysis unit, a hash table management unit, an entry sort execution unit, and a data page generation unit controlled by the bulk load control unit. The input file analysis unit analyzes the input file and reads data for each entry, and the hash table management unit manages the identification information of the entry in a hash table. The entry sort execution means manages the entries in an entry list, and the data page generation means generates the entry data in a database storage format.
[0034]
In this way, the entry sort execution means manages the entries in the entry list, that is, acquires the data contents of the entries in advance, so that the entry data included in the input file can be loaded efficiently and at high speed.
[0035]
  Claims6The invention described is the above claim.5In the described bulk loading system, the input file analyzing means reads the entry data of the input file for the first time, and the entry sort executing means performs a sorting process based on the entry list. The input file analysis means reads the input file entry data for the second time, and the hash table management means actually reads the entry read by the input file analysis means for the second time. ofidentificationA code is assigned, the identification information of the entry is registered in a hash table with the actual identification code, and a parent identification code is obtained for the entry representing a higher level concept. Based on the sorting process, the entry file entry data obtained by the second reading is generated in a database storage format.
[0036]
In this way, even if the input file consists of a plurality of entries, the data of the plurality of entries can be clustered and loaded together so that the search performance is improved.
[0037]
  Claims7The invention described is the above claim.6In the described bulk loading system, the input file analyzing means reads the first time and the second time of the data of all entries of the input file.
[0038]
In this way, even if the input file consists of a plurality of entries, the data of all the entries in the input file can be loaded in a batch.
[0039]
  Further, the claims of the present invention8In the described bulk loading method, the data processing apparatus includes a bulk load control unit, an input file analysis unit, a hash table management unit, a reference list management unit, and a data page generation unit controlled by the bulk load control unit. A bulk loading method comprising: the input file analyzing means analyzing the input file and generating data for each entry;Read the first time to read,The hash table management means manages identification information of the entry in a hash table, and the reference list management meansA reference list is generated based on a temporary identification code map and / or a temporary reference list having a temporary identification code for the entry of the input file, which is different from the actual identification code for the entry read by the input file analyzing means. The reference relation of the entry is managed by the reference list, the bulk load control unit generates an update list based on the reference list, and the input file analysis unit converts the data of the entry of the input file On the other hand, the second reading is performed, and the data page generation unit generates the input file entry data acquired in the second reading in the database storage format based on the update list.There is as a method.
[0040]
As described above, the present invention is also effective as an invention of a bulk loading method, and even when an input file is composed of a plurality of entries, data of a plurality of entries can be loaded at a time, and the data of the entries can be efficiently and It can be loaded at high speed.
[0041]
  Further, the claims of the present invention9In the described bulk loading method, the data processing apparatus includes a bulk load control unit, an input file analysis unit, a hash table management unit, an entry sort execution unit, and a data page generation unit controlled by the bulk load control unit. The input file analysis unit analyzes the input file and reads data for each entry, and the hash table management unit manages the identification information of the entry in a hash table. The entry sort execution means manages the entries in an entry list, further sorts the entries according to the identification information of the entries, and the data page generation means re-enters from the input file in the order of the sorted entries. Read the entry data into the database There a method for generating in storage format.
[0042]
As described above, the present invention is also effective as an invention of a bulk loading method, and even when an input file is composed of a plurality of entries, data of a plurality of entries is clustered so that search performance is improved. Bulk loading is possible.
[0043]
  Further, the claims of the present invention10The described bulk loading program parses the input file into a computer and generates data for each entry.Load the first time to loadProcessing, processing for managing the identification information of the entry in a hash table,A reference list is generated based on a temporary identification code map and / or a temporary reference list having a temporary identification code for the entry of the input file, which is different from the actual identification code for the entry read by the input file analyzing means. And manage the reference relationship of the entries in the reference list.Processing,A process for generating an update list based on the reference list, a process for performing a second reading on the entry data of the input file, and the reading obtained by a second reading based on the update list Generate input file entry data in database storage formatThe processing is executed.
[0044]
As described above, the present invention is also effective as an invention of a bulk loading program, and even when an input file is composed of a plurality of entries, data of a plurality of entries can be loaded at a time, and the data of the entries can be efficiently and It can be loaded at high speed.
[0045]
  Further, the claims of the present invention11The described bulk loading program is a computer that analyzes an input file and reads data for each entry, a process that manages identification information of the entry using a hash table, a management that manages the entry using an entry list, The process of sorting the entries according to the identification information of the entries, and the process of generating the data of the entries read again from the input file in the storage format of the database in the order of the sorted entries.
[0046]
As described above, the present invention is also effective as an invention of a bulk loading program, and even when an input file is composed of a plurality of entries, the data of a plurality of entries are clustered so that the search performance is improved. Bulk loading is possible.
[0047]
  Further, the claims of the present invention12The described bulk loading system includes an input device such as a keyboard, a data processing device, a storage device that stores information, a database management device that stores and manages information, a storage medium, and an output device such as a display device and a printing device. A bulk loading system comprising:10Or claim11The bulk loading program described in 1 is installed.
[0048]
In this way, by adopting a bulk loading system equipped with a bulk loading program, even if the input file consists of a plurality of entries, data of a plurality of entries can be loaded at once, or data of a plurality of entries can be loaded. , Clustering and batch loading to improve search performance.
[0049]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
First, a first embodiment of a bulk load system of the present invention will be described with reference to the drawings.
[0050]
"First embodiment of bulk loading system"
FIG. 1: has shown the schematic block diagram for demonstrating the basic composition of 1st embodiment of the bulk load system which concerns on this invention.
In the figure, a bulk load system includes an input device such as a keyboard 1, a data processing device that operates under program control, a storage device that stores information 3, a database management device 4 that stores and manages information, and a display device And an output device 5 such as a printing device.
[0051]
The storage device 3 includes an input file storage unit 31, a hash table storage unit 32, a temporary ID map storage unit 33, and a reference list storage unit 34.
[0052]
The input file storage unit 31 stores file data such as an LDIF file to be input.
The hash table storage unit 32 is a table used in the hash method, which is one of the data storage techniques, and stores a hash table storing arbitrary information using a character string or a number as a keyword.
[0053]
  A temporary ID storage unit 33 that stores a temporary identification code map (referred to as abbreviated as a temporary ID map 230, see FIG. 6) is provided with a temporary identification code (appropriately referred to as a temporary ID). (Abbreviated) and the actual identification code (abbreviated as real ID as appropriate)
  The temporary identification code is an identification code for an entry of the input file, and is an identification code different from the actual identification code set so as not to cause a collision between the actual identification code and the address. As shown in FIG. 6, when the real ID is 5 or 6, -2 and -3 are set as temporary IDs.
The temporary identification code map is a map composed of a temporary identification code and an actual identification code corresponding to the temporary identification code.
  As described above, by using the temporary ID in addition to the real ID, it is possible to appropriately use the hash method so as not to cause an address collision in the hash table.
[0054]
Furthermore, the reference list storage unit 34 stores a reference list 220 (see FIG. 6), that is, a list that represents the parent-child relationship of entries by using real IDs and temporary IDs assigned to the entries.
[0055]
The database management device 4 includes a data page storage unit 41 that holds information stored in the database in a page format.
In this way, by holding the information in the page format, when the information is output, it is not necessary to convert the information again according to the output format, and as a result, the processing speed can be increased.
[0056]
The data processing apparatus 2 includes a bulk load control unit 21, an input file analysis unit 22, a hash table management unit 23, a reference list management unit 24, and a data page generation unit 25 controlled by the bulk load control unit 21. It is as a configuration provided.
[0057]
When the bulk load control unit 21 receives an entry load request using the LDIF file from the user from the input device 1, the input file analysis unit 22, the hash table management unit 23, and the reference list management according to the request. The means 24 and the data page generation means 25 are controlled to perform bulk loading processing.
Further, when the bulk load process is completed, the bulk load control means 21 displays on the output device 4 that the bulk load process has been completed.
[0058]
Here, the input file analysis unit 22 reads and analyzes the LDIF file from the input file storage unit 31 in accordance with an entry load instruction from the bulk load control unit 21 and acquires information on entries included in the LDIF file.
[0059]
Similarly, the hash table management unit 23 creates a necessary hash table using the hash table storage unit 32 in accordance with an instruction from the bulk load control unit 21, calculates a hash value from the keyword value, and performs the above LDIF. Manages identification information of entries contained in files.
[0060]
Further, the reference list management means 24 creates a reference list 220 representing the parent-child relationship of the entries by using the temporary ID map storage section 33 and the reference list storage section 34, and in accordance with an instruction from the bulk load control means 21, Each entry with an ID or temporary ID is managed.
[0061]
Here, the reference list 220 corresponds to the real ID for the entry read by the input file analysis means 22, the parent identification code for the entry representing the superordinate concept (appropriately abbreviated as parent ID), and the entry representing the subordinate concept. It is preferable to have a configuration having a child identification code (abbreviated as a child ID as appropriate). In this way, the entry reference relationship is changed to a parent-child relationship based on the entry data (a combination of a superordinate concept and a subordinate concept). The reference list can be easily used.
[0062]
In addition, the reference list 220 may be configured to have a next identification code (appropriately abbreviated as “next ID” as appropriate) indicating the sibling relationship between the entries for the peer level entries. Can be represented by sibling relationships based on the entry data, and the reference list can be easily used.
[0063]
  In addition, the reference list 220 may be configured to be generated based on the temporary ID map 230 and / or the temporary reference list 210 having a temporary ID for the entry of the input file. Each time an entry is read, it can be stored in the temporary ID map 230 and / or the temporary reference list 210, and the operation of the system can be simplified.
  As shown in FIG. 6, the temporary reference list 210 is a list including a temporary ID, a parent ID, a child ID, and a next ID, and a reference for re-registering a parent-child relationship and the like from the temporary reference list to the reference list. It is a list that captures the list 220 (or can be said to be part of the reference list).
[0064]
Further, the data page generation unit 25 uses the reference list storage unit 34 in accordance with an instruction from the bulk load control unit 21 to configure the data of each entry in a database page format and store the data in the data page storage unit 41. .
[0065]
Next, the operation of the bulk load system configured as described above will be described with reference to the drawings.
FIG. 2: has shown the schematic flowchart figure for demonstrating operation | movement of the bulk load system concerning 1st embodiment.
[0066]
In FIG. 2, first, the bulk load control means 21 inputs the file name of the LDIF file input from the input device 1, and reads the corresponding file from the input file storage unit 31 by the input file analysis means 22. Data of one entry (referred to as one entry as appropriate) is acquired (step A1).
That is, the input file analyzing means 22 reads the data of the first entry from the input file.
[0067]
Subsequently, the hash table management unit 23 registers a hash table composed of the identification information and the real ID of the one entry in the hash table storage unit 32, and the reference list management unit 24 registers the temporary ID map storage unit 33. A reference list representing the reference relationship (parent-child relationship and sibling relationship) of the one entry is registered in the reference list storage unit 34 (step A2).
[0068]
Then, the bulk loading system in this embodiment checks whether this one entry is the last entry of the input file (step A3). If not, the process returns to step A1 and the same processing is performed for the next entry. Perform the process.
[0069]
  When the processing is completed up to the final entry in step A3, the reference list management unit 24 generates an update list (see FIG. 7) using the temporary ID map storage unit 33 and the reference list storage unit 34 (step 7). A4).
  The update list is a list including a real ID, a parent ID, a child ID, and a next ID. Based on the update list, the data page generation unit 25 stores the data of the entry of the input file acquired by the second reading. Generate in database storage format.
[0070]
Subsequently, the input file analyzing unit 22 reads the corresponding input file again from the input file storage unit 31 and acquires data of one entry (step A5). That is, the input file analysis means 22 reads the data of the second entry from the input file.
[0071]
Next, the data page generation unit 25 generates entry data from the acquired one entry (step A6), refers to the update list in the reference list storage unit 34, and sets the parent entry (entry having a parent relationship) in the entry data. Is registered (step A7), and a set of child entries (entries having child relations) is generated and registered in entry data (collection of entry data) (step A8).
That is, the data page generation unit 25 generates the input file entry data acquired in the second reading in the database storage format based on the update list.
[0072]
This entry data is configured in a page format, and is stored in the data page storage unit 41 when there is no free space on the page. That is, according to the bulk load system in the present embodiment, the entry data is collected by the data processing device 2. A collection of entry data can be loaded in a batch.
[0073]
Next, it is checked whether or not this entry is the last entry of the input file (step A9). If not, the process returns to step A5, and the same processing is performed for the next entry.
In step A9, when the process is completed up to the final entry, the entry data configured in the page format is stored in the data page storage unit 41 and displayed on the output device 5 to notify the end of the process.
[0074]
As described above, the reference list management unit 24 manages the reference relationship of the entries with the reference list 220, that is, acquires the reference relationship of the entries in advance, so that the data of the entries included in the input file can be efficiently and at high speed. Can load.
In addition, by using the update list, data of a plurality of entries can be loaded in a batch even when the input file includes a plurality of entries.
[0075]
Next, operations for registering the hash table and the reference list will be described with reference to the drawings.
FIG. 3 is a schematic flowchart for explaining the operation of registering the hash table and the reference list in the bulk load system according to the first embodiment.
In FIG. 3, the real ID is assigned to the entry (step B1), and it is checked whether or not the own entry (entry to which the real ID is assigned) is already registered in the hash table (step B2). Then, it is checked whether or not the parent entry (entry concept entry) is already registered in the hash table (step B6).
[0076]
Here, if not registered, a temporary ID is assigned to the parent entry (step B10), the parent-child relationship is registered in the temporary reference list (step B11), and the identification information and temporary ID are registered in the hash table (step B11). B12).
And it returns to step B6 again.
[0077]
Next, since the parent entry is registered with a temporary ID in step B6, the ID of the parent entry is acquired from the hash table (step B7), the parent-child relationship is registered in the reference list (step B8), and is identified in the hash table. Information and real ID are registered (step B9).
[0078]
If the parent entry has already been registered in step B6, the processes in steps B10 to B12 are omitted.
If the entry is already registered in step B2, the temporary ID and the real ID are registered in the temporary ID map (step B3), and the parent-child relationship is re-registered from the temporary reference list to the reference list (step B4). The identification information and real ID are re-registered in the hash table (step B5).
[0079]
As described above, the bulk load control unit 2 uses the temporary ID in addition to the real ID, and the hash table management unit 23 and the reference list management unit 24 further use the temporary ID map, the temporary reference list, the reference list, and the like. By using the update list, it is possible to appropriately use the hash method so that address collisions do not occur in the hash table.
[0080]
As described above, according to the first embodiment of the bulk load system, the input file entry is read twice in step A1 and step A5 as shown in FIG. As shown in FIG. 3, by acquiring the entry identification information and the reference relationship between entries, it is not necessary to refer to other entry data when generating entry data in step A6.
For this reason, it is not necessary to load data of entries one by one, that is, it is possible to realize a bulk load system capable of loading data of a plurality of entries at once.
[0081]
Further, the input file analyzing means 22 may be configured to read the first time and the second time of the data of all the entries of the input file, and in this case, the input file includes a plurality of entries. Even so, the data of all entries in the input file can be loaded at once.
[0082]
<Example>
Next, the operation of the first embodiment will be described using a specific example.
FIG. 4 is a schematic diagram showing an example of an LDIF file.
In the figure, the entry having the identifier “c = JP” is the root (highest level) of the directory, and this embodiment is an embodiment in which a lower entry is added under this entry.
[0083]
The “version: 1” in the first line represents the LDIF standard version.
The range delimited by blank lines from the next line is defined as “one entry data”, and this LDIF file contains ten “one entry data” (301 to 310).
In each “data of one entry”, a line beginning with “dn:” indicates identification information, and the other lines indicate attribute values (attribute data).
[0084]
When a directory is loaded by this LDIF file, a directory hierarchy as shown in FIG. 5 is constructed.
Here, the entry 100 is the root of the directory hierarchy and is registered in advance in the directory server. In contrast, the entries 101 to 110 are newly loaded entries.
[0085]
The names described in each of the entries 101 to 110 are relative distinguished names (RDN: Relative Distinguished Name). If the relative distinguished names of all the entries going back to the root are connected with a comma (,), they become the distinguished names of the entries. .
[0086]
When the LDIF file is read and the parent-child relationship between entries is acquired, for example, a temporary reference list 210, a reference list 220, and a temporary ID map 230 as shown in FIG. 6 are obtained.
[0087]
  In FIG. 6, the temporary ID map 230 indicates the real ID of the entry 100 that is the root.1When the real ID is assigned to the entry, it is increased from 1 in order, and when the temporary ID is assigned to the entry, it is decreased from -1.
  In this embodiment, -1, -2, and -3 are used as temporary IDs, and real IDs are used.1(301 in FIG. 4), 5 (305 in FIG. 4), and 6 (306 in FIG. 4).
[0088]
The temporary reference list 210 includes a temporary ID, a parent ID, a child ID, and a next ID. Except for a temporary integer that is a negative integer, IDs (1 to 10) that are positive integers are “data of one entry”. "(301 to 310 in FIG. 4).
[0089]
Further, the temporary reference list 210 and the reference list 220 are configured to have one child ID and one next ID with respect to the real ID, and in this way, even if there are two or more child IDs, Since sibling entries can be connected by ID, all child entries can be acquired by tracing the next ID in order from the child ID entry.
[0090]
The temporary ID map 230 is stored in the temporary ID map storage unit 33 in the format as shown in FIG. 6, and the temporary reference list 210 and the reference list 220 are similarly in the format as shown in FIG. It is stored in the storage unit 34.
[0091]
The reference list management means 24 uses the information stored in the temporary ID map storage unit 33 and the reference list storage unit 34 to generate the update list shown in FIG. 7 in step A4.
This update list can be generated by simply changing the temporary ID to the real ID using the temporary ID map 230 based on the reference list 220 of FIG.
[0092]
Next, how to read the update list will be described.
The real ID 1 (“1 entry data” 301 in FIG. 4) has no parent ID (root entry) and the child ID is 5 (“1 entry data” 305 in FIG. 4).
Further, the real ID 5 (“1 entry data” 305 in FIG. 4) displayed as 5 in the child ID has a parent ID of 1 (“1 entry data” 301 in FIG. 4) and a child ID of 2 (FIG. 4). 4 (“1 entry data” 302) and the next ID is 6 (“1 entry data” 306 in FIG. 4).
[0093]
Further, the real ID 6 displayed as 6 in the next ID has a parent ID of 1, a child ID of 4 (“1 entry data” 304 in FIG. 4), and a next ID of 7 (“1 entry of FIG. 4”). Data "307).
Further, the real ID 7 displayed as 7 in the next ID has a parent ID of 1, a child ID of 9 (“1 entry data” 309 in FIG. 4), and no next ID.
[0094]
The real ID 2 displayed as 2 in the child ID has a parent ID of 5, no child ID, and a next ID of 3 (“1 entry data” 303 in FIG. 4).
Further, the real ID 3 displayed as 3 in the next ID has a parent ID of 5 and has no child ID and next ID.
[0095]
The update list also represents the parent-child relationship in the same way for the other real IDs 4, 8, 9, 10 and easily and simply represents the parent ID, child ID, and next ID for any real ID. be able to.
[0096]
The data page generation unit 25 refers to the update list generated in the reference list storage unit 34, registers the parent entry in step A7, generates a child entry set in step A8, and registers it in the entry data. To do.
[0097]
As described above, according to the bulk load system of the present invention, a set of child entries is generated by referring to the update list and registered in the entry data. Therefore, a plurality of entries included in the input file are loaded in a batch. And the reading process can be speeded up.
[0098]
"Second embodiment of bulk loading system"
Next, a second embodiment of the bulk load system of the present invention will be described in detail with reference to the drawings.
FIG. 8: has shown the schematic block diagram for demonstrating the basic composition of 2nd embodiment of the bulk load system which concerns on this invention.
In the figure, in the bulk load system of the second embodiment of the present invention, the data processing device 6 includes a bulk load control means 21, an input file analysis means 22, a hash table management means 23, an entry sort execution means 26, The data page generation means 25 is provided, and the storage device 7 includes an input file storage unit 31, a hash table storage unit 32, and an entry sort storage unit 35.
[0099]
As shown in FIG. 10, the entry sort storage unit 35 stores the identification name of each entry and the number of lines in the LDIF file as an entry list.
Further, the entry sort execution unit 26 registers the identification name and the number of rows of each entry in the entry list of the entry sort storage unit 35 in accordance with an instruction from the bulk load control unit 21, and uses the identification name as a key. Sort.
Other configurations are the same as those of the bulk load system in the first embodiment.
[0100]
Next, the operation of the bulk load system configured as described above will be described with reference to the drawings.
FIG. 9: has shown the schematic flowchart figure for demonstrating operation | movement of the bulk load system concerning 2nd embodiment.
FIG. 10 shows a table for explaining an entry list constructed by reading an LDIF file.
[0101]
In FIG. 9, first, the bulk load control means 21 inputs the file name of the LDIF file given from the input device 1, the corresponding file is read from the input file storage unit 31 by the input file analysis means 22, and 1 The entry data is acquired (step A1).
Subsequently, the bulk load control means 21 registers the entry identification name and the number of lines on the file in the entry sort storage section 35 by the entry sort execution means 26 (step C1).
[0102]
Next, it is checked whether or not this entry is the last entry of the input file (step A3). If it is not the last entry, the process returns to step A1.
When the processing is completed up to the last entry in step A3, the entry sort execution means 26 sorts the entries registered in the entry sort storage unit 35 by the identification name (step C2).
[0103]
The entry sort execution unit 26 acquires one entry from the sorted entry list in the entry sort storage unit 35 (step C3), assigns a real ID to the entry (step B1), and subsequently the hash table management unit 23. Registers the identifier and real ID in the hash table of the hash table storage unit 32 (step B9), and acquires the ID of the parent entry from the hash table (step B7).
[0104]
The data page generation means 25 generates entry data (step A6), registers a parent entry in the generated entry data (step A7), and registers it as a child entry in the parent entry data (step C4).
[0105]
Next, it is checked whether or not this entry is the last entry in the entry list (step A9). If it is not the last entry, the process returns to step C3.
In step A9, when the process is completed up to the final entry, it is displayed on the output device 5 to notify the end of the process.
[0106]
As described above, in the bulk load system according to the present embodiment, the entry sort execution unit manages the entries in the entry list, that is, sorts all the entries described in the input file by the identification name, and closes them on the directory hierarchy. Since certain entries can be registered together, it is possible to reduce the possibility of performing unnecessary processing such as rereading registered entries.
In addition, since the data contents of the entries are acquired in advance, even if the input file is composed of a plurality of entries, the data of the plurality of entries can be clustered and loaded together so as to improve the search performance.
[0107]
<Example>
Next, the operation of the second embodiment will be described using a specific example.
As in the example of the first embodiment, it is assumed that the LDIF file shown in FIG. 4 is an input file and a directory hierarchy as shown in FIG. 5 is constructed.
[0108]
The LDIF file is read, the identification name and the number of lines in the LDIF file are registered in the entry list for each entry, and the entry list as a result of sorting by the identification name is as shown in FIG.
[0109]
The entry sort execution means 26 registers entries in the entry list in the entry sort storage unit 35 in step C1, and sorts the entries registered in step C2 by identification name.
In performing this sort, the entry sort execution means 26 can cluster the entry data in consideration of the directory hierarchy, so that the search performance of the directory service can be improved.
[0110]
In step C3, one entry is extracted from the entry list, and the information of the corresponding entry can be read from the input file storage unit 31 by the input file analysis means 22 using the number of lines of the entry.
[0111]
In this way, the bulk loading system in the present embodiment sorts all entries described in the input file by the identification name, and registers the entries in the vicinity on the directory hierarchy to register a plurality of input files. Even in the case of entries, the data of a plurality of entries can be clustered and loaded together so as to improve the search performance.
[0112]
“First Embodiment of Bulk Loading Method”
  The present invention is also effective as a bulk loading method, and the above-described FIG., 3The bulk loading method according to the first embodiment will be described using the schematic flowchart shown in FIG.
  The bulk loading method in the present embodiment includes a data processing device controlled by a bulk loading control unit, a bulk loading control unit, an input file analysis unit, a hash table management unit, a reference list management unit, and a data page generation unit. The input file analysis means analyzes the input file and outputs data for each entry.Read the first time to read,That is, data of one entry is acquired (step A1).
[0113]
  Next, the hash table management means manages the identification information of the entry in the hash table, and the reference list management meansAs shown in FIG. 3, on the basis of a temporary identification code map and / or a temporary reference list having a temporary identification code for an entry of the input file, which is different from the actual identification code for the entry read by the input file analysis means. Generate a reference list and manage the reference relationship of entries in the reference listTo do. That is, a hash table is registered and a reference list is registered (step A2).
[0114]
  next,The bulk load control means generates an update list based on the reference list (step A4), and then the input file analysis means reads the data of the entry of the input file for the second time (step A4). A5), andThere is a method in which the data page generation means generates the data of the entry read again from the input file in the database storage format based on the update list generated from the reference list (steps A6, A7 and A8).
  That is, the data page generation unit generates the data of the input file entry acquired in the second reading in the database storage format based on the update list.
[0115]
As described above, the present invention is also effective as an invention of a bulk loading method, and even when an input file is composed of a plurality of entries, data of a plurality of entries can be loaded at a time, and the data of the entries can be efficiently and It can be loaded at high speed.
[0116]
"Second embodiment of bulk loading method"
The present invention is also effective as a bulk loading method, and the bulk loading method according to the second embodiment will be described using the schematic flowchart shown in FIG. 9 described above.
The bulk loading method according to the present embodiment includes a data processing device controlled by a bulk load control unit, a bulk load control unit, an input file analysis unit, a hash table management unit, an entry sort execution unit, and a data page generation unit. The input file analysis means analyzes the input file and reads data for each entry, that is, acquires data of one entry (step A1).
[0117]
Next, the hash table management means manages entry identification information using a hash table, and the entry sort execution means manages entries using an entry list. That is, the identification information of each entry is registered in the entry list (step C1).
Then, each entry is sorted by the entry identification information (step C2).
[0118]
Next, there is a method in which the data page generation means generates the data of the entry read again from the input file in the storage format of the database in the sorted entry order (steps C3, A6, A7 and C4).
That is, the data page generation unit generates the data of the entry of the input file acquired by the second reading in the database storage format based on the entry list.
[0119]
As described above, the present invention is also effective as an invention of a bulk loading method, and even when an input file is composed of a plurality of entries, data of a plurality of entries is clustered so that search performance is improved. Bulk loading is possible.
[0120]
"First embodiment of bulk loading program"
  The present invention is also effective as a bulk load program, and the bulk load program according to the first embodiment will be described with reference to the schematic flowchart shown in FIG.
  In this embodiment, the bulk loading program first analyzes the input file into the computer and stores the data for each entry.Load the first time to loadThe process (step A1) is executed.
[0121]
  Next, the bulk load program causes the computer to manage entry identification information using a hash table, and to manage entry reference relationships using a reference list.A reference list is generated based on a temporary identification code map and / or a temporary reference list having a temporary identification code for an entry of the input file, which is different from the actual identification code for the entry read by the input file analysis means, For managing reference relationships in a reference list(Step A2) is executed.
Subsequently, based on the reference list, a process for generating an update list (step A4) is executed, and further, a process for performing a second reading on the entry data of the input file (step A5) is executed. .
[0122]
  Subsequently, the bulk loading program causes the computer to execute an update list generated from the reference list.Generate the input file entry data obtained in the second reading in the database storage formatThe process (steps A6, A7 and A8) is executed.
[0123]
As described above, the present invention is also effective as an invention of a bulk loading program, and even when an input file is composed of a plurality of entries, data of a plurality of entries can be loaded at a time, and the data of the entries can be efficiently and It can be loaded at high speed.
[0124]
"Second embodiment of bulk loading program"
The present invention is also effective as a bulk load program, and the bulk load method according to the second embodiment will be described using the schematic flowchart shown in FIG. 9 described above.
The bulk load program according to the present embodiment first causes the computer to analyze the input file and execute a process of reading data for each entry (step A1).
[0125]
Next, the bulk load program manages the entry identification information in the computer using a hash table and manages the entry in the entry list, that is, the process of registering the identification information of each entry in the entry list (step C1). Is executed.
Next, the bulk load program causes the computer to execute a sort process (step C2) for each entry based on the identification information of the entry.
[0126]
Next, the bulk load program causes the data page generation means to generate, in the storage order of the database, the data of the entries read again from the input file in the order of the sorted entries in the computer (steps C3, A6, and C6). A7 and C4) are executed.
That is, the data page generation unit generates the data of the entry of the input file acquired by the second reading in the database storage format based on the entry list.
[0127]
As described above, the present invention is also effective as an invention of a bulk loading program, and even when an input file is composed of a plurality of entries, the data of a plurality of entries are clustered so that the search performance is improved. Bulk loading is possible.
[0128]
The bulk load program can be stored in a computer-readable recording medium such as an external storage device and a portable recording medium in addition to being stored in the ROM of the computer.
Here, the external storage device refers to a storage expansion device that incorporates a recording medium such as a magnetic disk and is externally connected to the data processing device. On the other hand, the portable recording medium is a recording medium that can be mounted on a recording medium driving device (drive device) and can be carried, for example, a CD-ROM, a flexible disk, a memory card, a magneto-optical disk, and the like. .
[0129]
The program stored in the recording medium is loaded into the RAM of the computer and executed by the CPU. By this execution, each function of the above-described bulk load system is realized.
Furthermore, when a control program is loaded by a computer, the control program held by another computer can be downloaded to its own RAM or external storage device using a communication line.
[0130]
Further, the bulk load system of the present invention may be configured such that the data processing apparatus is mounted with the bulk load program according to claim 12 or claim 13, and the bulk load system will be described with reference to the drawings. .
[0131]
“Third embodiment of bulk loading system”
FIG. 11 is a schematic block diagram for explaining the basic configuration of the third embodiment of the bulk load system according to the present invention.
In the figure, the bulk loading system includes an input device 1 such as a keyboard, a data processing device 9, a storage device 3 for storing information, a database management device 4 for storing and managing information, a storage medium 8, and a display device. An output device 5 such as a printing device is used.
The recording medium 8 may be a magnetic disk, a semiconductor memory, or other recording medium.
[0132]
The bulk load program is read from the recording medium 8 into the data processing device 9 and controls the operation of the data processing device 9.
The data processing device 9 is configured to execute the following processing under the control of the bulk load program, that is, the same processing as that performed by the data processing devices 2 and 6 in the first and second embodiments.
Other configurations are the same as those of the bulk load system in the first or second embodiment.
[0133]
  When a bulk load request is given from the input device 1, the bulk load system configured as described above reads the input file specified by the request content from the input file storage unit 31 in the storage device 3.10When the described bulk loading program is installed, the identification name and ID of each entry are registered in the hash table storage unit 32, an update list is generated using the temporary ID map storage unit 33 and the reference list storage unit 34, Data of all entries is generated and stored in the data page storage unit 41 in the database management apparatus 4 in the database page format.
  The end of these processes is displayed on the output device 5.
  The above claims11Of course, when the described bulk load program is installed, the effect of such a program is exhibited.
[0134]
As described above, the bulk load system of the present embodiment is a bulk load system equipped with a bulk load program, so that even when an input file is composed of a plurality of entries, data of a plurality of entries can be loaded at once. Alternatively, data of a plurality of entries can be clustered and loaded in a batch so that search performance is improved.
[0135]
Although the present invention has been described as a bulk load system, it is needless to say that the same effect can be achieved by a bulk load method, a bulk load program, and a bulk load system equipped with this bulk load program.
[0136]
【The invention's effect】
As described above, according to the bulk loading system, the bulk loading method, and the bulk loading program of the present invention, the input file is read twice, the entry reference relationship is canceled by the first reading, and the second reading is performed. Since the data is generated by using the reference relationship, a plurality of entry data can be loaded at a time from the input file without loading the data for each entry.
[0137]
Further, according to the bulk load system, bulk load method, and bulk load program of the present invention, the entry data is sorted according to the identification information of the entries and the entry data is generated in the sorted order. Can be clustered.
[Brief description of the drawings]
FIG. 1 is a schematic block diagram for explaining a basic configuration of a first embodiment of a bulk load system according to the present invention.
FIG. 2 is a schematic flowchart for explaining the operation of the bulk load system according to the first embodiment.
FIG. 3 is a schematic flowchart for explaining an operation of registering a hash table and a reference list in the bulk load system according to the first embodiment.
FIG. 4 is a schematic diagram showing an example of an LDIF file.
FIG. 5 is a schematic diagram for explaining a directory hierarchy when an LDIF file is loaded.
FIG. 6 shows a table for explaining a temporary ID map, a temporary reference list, and a reference list constructed by reading an LDIF file.
FIG. 7 shows a table for explaining an update list of the present invention.
FIG. 8 is a schematic block diagram for explaining a basic configuration of the second embodiment of the bulk load system according to the present invention.
FIG. 9 is a schematic flowchart for explaining the operation of the bulk load system according to the second embodiment.
FIG. 10 is a table for explaining an entry list constructed by reading an LDIF file.
FIG. 11 is a schematic block diagram for explaining a basic configuration of a third embodiment of a bulk load system according to the present invention.
[Explanation of symbols]
1 Input device
2,6,9 data processing device
3,7 storage device
4 Database management device
5 Output device
21 Bulk load control means
22 Input file analysis means
23 Hash table management means
24 Reference list management means
25 Data page generation means
26 Entry sort execution means
31 Input file storage
32 Hash table storage
33 Temporary ID map storage
34 Reference list storage
35 entry sort storage
41 Data page storage
8 Recording media
100-110 Directory hierarchy example entries
210 Temporary reference list
220 Reference list
230 Temporary ID map
301-310 1 entry data

Claims (12)

A data processing apparatus is a bulk load system including a bulk load control unit and an input file analysis unit, a hash table management unit, a reference list management unit, and a data page generation unit controlled by the bulk load control unit. There,
The input file analyzing means analyzes the input file and performs a first reading to read data for each entry ,
The hash table management means manages the identification information of the entry in a hash table;
In the temporary identification code map and / or the temporary reference list, the reference list management means has a temporary identification code for the entry of the input file different from the actual identification code for the entry read by the input file analysis means. Based on this, a reference list is generated, and the reference relationship of the entry is managed in the reference list.
The bulk load control means generates an update list based on the reference list,
The input file analyzing means performs a second reading on the input file entry data,
The bulk load system, wherein the data page generation unit generates data of the entry of the input file acquired by the second reading in a database storage format based on the update list . The reference list is
An actual identification code for the entry read by the input file analysis means;
A parent identification code for the entry representing a superordinate concept;
The bulk load system according to claim 1, further comprising: a child identification code for the entry representing a subordinate concept. The reference list is
3. The bulk load system according to claim 2, further comprising: a next identification code indicating a sibling relationship between the entries for the entry at the peer level. The bulk loading according to any one of claims 1 to 3, wherein the input file analyzing means performs first and second reading on data of all entries of the input file. system. A data processing apparatus is a bulk load system including a bulk load control unit, an input file analysis unit, a hash table management unit, an entry sort execution unit, and a data page generation unit controlled by the bulk load control unit. And
The input file analyzing means analyzes the input file, reads data for each entry,
The hash table management means manages the identification information of the entry in a hash table;
The entry sort execution means manages the entries in an entry list;
The bulk load system, wherein the data page generation means generates data of the entry in a database storage format. The input file analyzing means performs a first reading on the data of the input file entry,
The entry sort execution means performs a sort process based on the entry list,
The input file analyzing means performs a second reading on the input file entry data,
The hash table management unit assigns an actual identification code to the entry read by the input file analysis unit a second time, and registers the identification information of the entry in the hash table with the actual identification code, Further, a parent identification code is obtained for the entry representing the superordinate concept,
The bulk load according to claim 5, wherein the data page generation unit generates data of the entry of the input file acquired by the second reading in a database storage format based on the sorting process. system. The bulk load system according to claim 6, wherein the input file analysis unit performs first and second readings on data of all entries of the input file. A data processing apparatus is a bulk load method comprising a bulk load control means and an input file analysis means, a hash table management means, a reference list management means, and a data page generation means controlled by the bulk load control means. And
The input file analyzing means analyzes the input file and performs a first reading to read data for each entry ,
The hash table management means manages the identification information of the entry in a hash table;
In the temporary identification code map and / or the temporary reference list, the reference list management means has a temporary identification code for the entry of the input file different from the actual identification code for the entry read by the input file analysis means. Based on this, a reference list is generated, and the reference relationship of the entry is managed in the reference list.
The bulk load control means generates an update list based on the reference list,
The input file analyzing means performs a second reading on the input file entry data,
The bulk loading method, wherein the data page generation unit generates the data of the entry of the input file acquired by the second reading in a database storage format based on the update list . A data processing apparatus is a bulk load method comprising a bulk load control means, an input file analysis means, a hash table management means, an entry sort execution means, and a data page generation means controlled by the bulk load control means. And
The input file analyzing means analyzes the input file, reads data for each entry,
The hash table management means manages the identification information of the entry in a hash table;
The entry sort execution means manages the entries in an entry list, and further sorts the entries according to the identification information of the entries,
The bulk loading method, wherein the data page generation unit generates data of an entry read again from an input file in the order of the sorted entries in a database storage format. Processing that causes the computer to analyze the input file and read the data for each entry for the first time ,
A process of managing identification information of the entry in a hash table;
A reference list is generated based on a temporary identification code map and / or a temporary reference list having a temporary identification code for the entry of the input file, which is different from the actual identification code for the entry read by the input file analyzing means. A process of managing the reference relationship of the entry in the reference list ;
A process for generating an update list based on the reference list;
A process for performing a second reading on the entry data of the input file;
A process of generating entry data of the input file obtained in the second reading based on the update list in a database storage format ;
A bulk load program to execute. Processing to analyze the input file to the computer and read the data for each entry,
A process of managing identification information of the entry in a hash table;
Managing the entries in an entry list, and further sorting the entries according to the identification information of the entries;
Processing for generating the data of the entries read again from the input file in the storage format of the database in the order of the sorted entries;
A bulk load program to execute. A bulk loading system comprising an input device such as a keyboard, a data processing device, a storage device for storing information, a database management device for storing and managing information, a storage medium, and an output device such as a display device and a printing device. And
A bulk loading system comprising the bulk loading program according to claim 10 or 11 .

Images