CN110347673A - Data file loading method, device, computer equipment and storage medium - Google Patents
Data file loading method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110347673A CN110347673A CN201910462330.5A CN201910462330A CN110347673A CN 110347673 A CN110347673 A CN 110347673A CN 201910462330 A CN201910462330 A CN 201910462330A CN 110347673 A CN110347673 A CN 110347673A
- Authority
- CN
- China
- Prior art keywords
- data
- file
- parameter
- tables
- sqlloader
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011068 loading method Methods 0.000 title claims abstract description 55
- 238000004590 computer program Methods 0.000 claims description 17
- 230000008676 import Effects 0.000 claims description 17
- 230000015654 memory Effects 0.000 claims description 11
- 238000000034 method Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 15
- 241000208340 Araliaceae Species 0.000 description 3
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 3
- 235000003140 Panax quinquefolius Nutrition 0.000 description 3
- 238000005194 fractionation Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 235000008434 ginseng Nutrition 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 239000003292 glue Substances 0.000 description 1
- 238000012905 input function Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a kind of data file loading method, device, computer equipment and storage medium, the present invention includes creating the tables of data of specified table structure；Read control file and data file；Obtain the parameter configured for data assembly tool；Judge whether the data format of data file matches with the table structure of tables of data；If the data format of data file and the table structure of tables of data match, judge whether the quantity of documents of read data file is greater than the file load limit value of the parameter of data assembly tool；If the file that the quantity of documents of data file is greater than the parameter loads limit value, limit value is loaded according to the file, data file is split, obtain multiple subdata files；Multiple subprocess of log-on data assembly tool carry out loaded in parallel to multiple subdata files, and multiple subdata files are loaded onto the tables of data.The present invention is based on the loaded in parallel that data file is realized in data processing, and improve the efficiency of file load.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of data file loading methods, device, computer equipment
And storage medium.
Background technique
Currently, the load for required variable files in account management system decision, traditional data file load
Mode uses data loading tool to be loaded, but traditional data loading tool is not supported between file internal and file simultaneously
Row load, in addition, the data volume that traditional data file loading method is limited to mass data itself is big, contains much information, produces in real time
Characteristics, the simple load data such as raw inevitably cause loading efficiency inefficient.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of data file loading method, device, computer equipment and storage
Medium, can be realized the loaded in parallel of data file, and improve the efficiency of file load.
On the one hand, the embodiment of the present invention provides a kind of data file loading method, and described method includes following steps:
Creation needs to import the tables of data of the specified table structure of data；
Read control file and data file；
Obtain the parameter configured for data assembly tool SQLLoader；
Judge whether the data format of the data file matches with the table structure of the tables of data created；
If the data format of the data file and the table structure of the tables of data created match, according to the number configured
According to the parameter of assembly tool SQLLoader, judge whether the quantity of documents of read data file is greater than configured data
The file of the parameter of assembly tool SQLLoader loads limit value；
If parameter of the quantity of documents of read data file greater than configured data assembly tool SQLLoader
File loads limit value, loads limit value to the data according to the file of the parameter of the data assembly tool SQLLoader
File is split, and multiple subdata files are obtained；
According to the multiple subdata file, start multiple subprocess of the data assembly tool SQLLoader to institute
It states multiple subdata files and carries out loaded in parallel, the multiple subdata file is loaded onto the tables of data.
On the other hand, the embodiment of the present invention provides a kind of data file loading device, and described device includes:
Creating unit needs to import the tables of data of the specified table structure of data for creating；
Reading unit, for reading control file and data file；
Acquiring unit, for obtaining the parameter for being directed to data assembly tool SQLLoader and being configured；
First judging unit, for judge the data file data format whether the table knot with the tables of data created
Structure matches；
Second judgment unit, if the table structure phase of data format and the tables of data created for the data file
Match, according to the parameter of the data assembly tool SQLLoader configured, judge read data file quantity of documents whether
The file of parameter greater than the data assembly tool SQLLoader configured loads limit value；
Split cells, if the quantity of documents for read data file is greater than configured data assembly tool
The file of the parameter of SQLLoader loads limit value, is loaded according to the file of the parameter of the data assembly tool SQLLoader
Limit value splits the data file, obtains multiple subdata files；
Start-up loading unit, for starting the data assembly tool SQLLoader according to the multiple subdata file
Multiple subprocess loaded in parallel is carried out to the multiple subdata file, the multiple subdata file is loaded onto described
Tables of data.
Another aspect the embodiment of the invention also provides a kind of computer equipment, including memory, processor and is stored in
On the memory and the computer program that can run on the processor, which is characterized in that described in the processor executes
Data file loading method as described above is realized when computer program.
It is described computer-readable to deposit in another aspect, the embodiment of the invention also provides a kind of computer readable storage medium
Storage media is stored with one or more than one computer program, and the one or more computer program can be by one
Or more than one processor executes, to realize data file loading method as described above.
As seen from the above, a kind of data file loading method of the embodiment of the present invention, device, computer equipment and storage are situated between
Matter needs to import the tables of data of the specified table structure of data by creation；Read control file and data file；It obtains for number
The parameter configured according to assembly tool SQLLoader；Judge the data file data format whether with the data that are created
The table structure of table matches；If the data format of the data file and the table structure of the tables of data created match, according to
The parameter of the data assembly tool SQLLoader configured, judges whether the quantity of documents of read data file is greater than institute
The file of the parameter of the data assembly tool SQLLoader of configuration loads limit value；If the number of files of read data file
The file of parameter of the amount greater than configured data assembly tool SQLLoader loads limit value, according to the data assembler
The file load limit value for having the parameter of SQLLoader splits the data file, obtains multiple subdata files；Root
According to the multiple subdata file, start multiple subprocess of the data assembly tool SQLLoader to the multiple subnumber
Loaded in parallel is carried out according to file, the multiple subdata file is loaded onto the tables of data.It can be realized using the present invention
The loaded in parallel of data file, and improve the efficiency of file load.
Detailed description of the invention
Technical solution in order to illustrate the embodiments of the present invention more clearly, below will be to needed in embodiment description
Attached drawing is briefly described, it should be apparent that, drawings in the following description are some embodiments of the invention, general for this field
For logical technical staff, without creative efforts, it is also possible to obtain other drawings based on these drawings.
Fig. 1 is a kind of application scenarios schematic diagram of data file loading method provided in an embodiment of the present invention；
Fig. 2 is a kind of schematic flow diagram of data file loading method provided in an embodiment of the present invention；
Fig. 3 is a kind of another schematic flow diagram of data file loading method provided in an embodiment of the present invention；
Fig. 4 is a kind of another schematic flow diagram of data file loading method provided in an embodiment of the present invention；
Fig. 5 is a kind of another schematic flow diagram of data file loading method provided in an embodiment of the present invention；
Fig. 6 is a kind of schematic block diagram of data file loading device provided in an embodiment of the present invention；
Fig. 7 is a kind of another schematic block diagram of data file loading device provided in an embodiment of the present invention；
Fig. 8 is a kind of another schematic block diagram of data file loading device provided in an embodiment of the present invention；
Fig. 9 is a kind of another schematic block diagram of data file loading device provided in an embodiment of the present invention；
Figure 10 is a kind of another schematic block diagram of data file loading device provided in an embodiment of the present invention；
Figure 11 is a kind of structure composition schematic diagram of computer equipment provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
It should be appreciated that ought use in this specification and in the appended claims, term " includes " and "comprising" instruction
Described feature, entirety, step, operation, the presence of element and/or component, but one or more of the other feature, whole is not precluded
Body, step, operation, the presence or addition of element, component and/or its set.
It is also understood that mesh of the term used in this description of the invention merely for the sake of description specific embodiment
And be not intended to limit the present invention.As description of the invention and it is used in the attached claims, unless on
Other situations are hereafter clearly indicated, otherwise " one " of singular, "one" and "the" are intended to include plural form.
It will be further appreciated that the term "and/or" used in description of the invention and the appended claims is
Refer to any combination and all possible combinations of one or more of associated item listed, and including these combinations.
Referring to Fig. 1, Fig. 1 is a kind of application scenarios signal of data file loading method provided in an embodiment of the present invention
Figure, the application scenarios include:
(1) server, server are used to provide the back-end services of data transmission.Server is a kind of computer equipment, can
Think single server or server cluster, or Cloud Server, or be special web page server, it receives external
The access of terminal is connect by cable network or wireless network with terminal.
(2) terminal, terminal shown in Fig. 1 include terminal 1, terminal 2 and terminal 3, and the terminal is by access server, from clothes
Target data is obtained on business device, acquired data file is loaded onto the database data table in terminal at the terminal.Institute
Stating terminal can be the electronic equipments such as smart phone, smartwatch, laptop, tablet computer or desktop computer, terminal
Server is accessed by cable network or wireless network.
Referring to Fig. 2, Fig. 2 is a kind of schematic flow diagram of data file loading method of the embodiment of the present invention, such as Fig. 2 institute
Show, this approach includes the following steps S101~S107.
Step S101, creation need to import the tables of data of the specified table structure of data.
In embodiments of the present invention, using sql like language (Structured Query Language, structuralized query language
Speech) create the tables of data for needing to import the specified table structure of data, wherein and the tables of data created is stored in target database
In, and the data for needing to import are loaded in the way of loaded in parallel by the target database；Specifically, in the present embodiment
The tables of data of specified table structure is created using the CREATE TABLE sentence of sql like language；Wherein, specified table structure refers to basis
The data imported are determined, which has determining specified table name, column name, data type and data class
The digit of type, such as specified table name are known as test, column name host, data type VARCHAR2, the digit of data type
It is 30.
Specifically, as shown in figure 3, the step S101 includes the following steps S201~S202:
Step S201 determines the table name, column name, data type sum number of the tables of data of specified table structure to be created
According to the maximum number of digits of type.
Step S202 is used according to the maximum number of digits of identified table name, column name, data type and data type
The CREATE TABLE sentence of sql like language creates the tables of data of the specified table structure.
In embodiments of the present invention, the number of the specified table structure is created using the CREATE TABLE sentence of sql like language
According to table, it is as follows to create the data tableau format: ((data type is most for 1 data type of column name for CREATE TABLE table name
Big digit), 2 data type of column name (maximum number of digits of data type), the column name 3 data type (dominant bit of data type
Number) ...).
Step S102 reads control file and data file.
In embodiments of the present invention, the control file is to be directed into Oracle number for the data controlled in data file
According in library, the data file refers to the data file that needs import, and includes the data of importing in need in the data file；
Wherein, the control file can add relevant control parameter according to their own needs, under normal circumstances, the control text
Part is made of following a few line program codes, comprising: the first row, LOAD DATA indicate that SQLLoader needs to load data；Second
Row, INFILE* indicate that data are wherefrom come；The third line, INTO TABLE indicate data to lead where；Fourth line, FIELDS
TERMINATED BY indicates separated between data with what symbol；Fifth line, " (*, *) " indicate that data need in what order
Write column the inside；6th row, BEGINDATA be indicate control file it is subsequent be all data file data；In ordinary circumstance
Under, only need the file address of configuration data file, when executing the file address for reading data file, lead to behind control file
The file address for crossing configured data file obtains the data of data file, which can be by data assembly tool
SQLLoader is generated, and can also be generated by other terminals, and the data file is added by data assembly tool SQLLoader
The tables of data being loaded onto target database.
Step S103 obtains the parameter configured for data assembly tool SQLLoader.
In embodiments of the present invention, the ROWS parameter that data assembly tool SQLLoader is arranged is 10, indicates every 10 line number
It is primary according to submitting；The BINDSIZE parameter that data assembly tool SQLLoader is arranged is 600Bytes, indicates to submit record every time
Buffer area maximum value；The READ BUFFER parameter that data assembly tool SQLLoader is arranged is 200Bytes, indicates to read
Take the size of buffer area.Wherein, the record number read every time is necessarily less than the value of ROWS parameter, and size must not exceed
The value of BINDSIZE parameter.
It should be noted that the parameter of SQLLoader further includes DIRECT parameter, PARALLEL parameter, ERROR parameter
Deng；Wherein, DIRECT parameter is loaded for setting SQLLoader using directapath, is set as true to the property of can choose；
PARALLEL parameter is set as true to the property of can choose in the present embodiment for setting SQLLoader using loaded in parallel.
Step S104, judge the data file data format whether the table structure phase with the tables of data created
Match.
In embodiments of the present invention, the data format of the data file needs and the data that are created in target database
The table structure of table matches, so that the data in data file are loaded onto the tables of data for meeting data format, if data file
In the data formats of data mismatched with the table structure of tables of data created, then the data in data file will be unable to load
To in corresponding tables of data；Wherein, judge the data file data format whether the table structure with the tables of data created
The standard to match is: first, the table structure pair of tables of data in the data type and target database of the data in data file
The data type answered；Second, the digit of the data type of data data corresponding with the table structure of tables of data in target database
The maximum number of digits of type.If the data type of the data in data file is corresponding with the table structure of tables of data in target database
Data type matches, and the digit of the data type of data is no more than the corresponding number of table structure of tables of data in target database
According to the maximum number of digits of type, then the data format of the data file and the table structure of the tables of data created match.
Specifically, as shown in figure 4, the step S104 includes the following steps S301~S303:
Step S301 analyzes the data format of data in the data file.
In embodiments of the present invention, the data format of data includes that data type, the corresponding digit of data type, data are big
It is small etc.；Wherein, data type may include Boolean type (boolean), byte type (byte), integer (int), short
(short), long (long) and single-precision floating point type (float) etc., the corresponding digit of data type can be according to data sheets
The size self-defining of body, under normal circumstances, the data value range for byte type are 0-255byte.
Step S302 determines the data type and data class of data according to the data format of data in the data file
Type digit.
In embodiments of the present invention, by the data in analysis data file, the specific data format of data is obtained, is passed through
Acquired data format so that it is determined that data attribute, data type (such as boolean, byte or int including data
Deng) and identified data type digit, for example, the digit accounted for for the data type of short is 16, long
32 and integer 16 etc..
Step S303, if the data type of identified data is corresponding with the data type of the tables of data created and institute
The data type digit of determining data is corresponding with the maximum number of digits of the tables of data created, determines the number of the data file
Match according to format and the table structure of the tables of data created.
In embodiments of the present invention, the data type of identified data is opposite with the data type of the tables of data created
The data type of data determined by should referring to is identical as the data type of the tables of data created, the number of identified data
According to type digit it is corresponding with the maximum number of digits of the tables of data created refer to determined by data data type digit with
The maximum number of digits of the tables of data created is identical or the data type digit of identified data is no more than created data
The maximum number of digits of table；The data format of the data file needs the table structure phase with the tables of data created in target database
Matching, so that the data in data file are loaded onto the tables of data for meeting data format, if the digit of the data type of data
No more than the maximum number of digits of the corresponding data type of table structure of tables of data in target database, then data of the data file
Format and the table structure of the tables of data created match.
Step S105, if the data format of the data file and the table structure of the tables of data created match, according to
The parameter of the data assembly tool SQLLoader configured, judges whether the quantity of documents of read data file is greater than institute
The file of the parameter of the data assembly tool SQLLoader of configuration loads limit value.
In embodiments of the present invention, the file load limit value refers to the parameter of data assembly tool SQLLoader
The greatest measure that data file can achieve can be loaded；If the quantity of documents of read data file is less than configured number
Limit value is loaded according to the file of the parameter of assembly tool SQLLoader, then illustrates to load data in this partial document data not
It will affect the loading performance of data assembly tool SQLLoader；It is configured if the quantity of documents of read data file is greater than
Data assembly tool SQLLoader parameter file load limit value, then illustrate to load the number in this partial document data
According to the loading performance for having influenced data assembly tool SQLLoader, the quantity of data file of the reason to be loaded is tight
It is again more than the file load limit value of the parameters dictate of data assembly tool SQLLoader, therefore, in order to improve data
Loading efficiency, improve system function optimization, need to split data file.
Step S106, if the quantity of documents of read data file is greater than configured data assembly tool
The file of the parameter of SQLLoader loads limit value, is loaded according to the file of the parameter of the data assembly tool SQLLoader
Limit value splits the data file, obtains multiple subdata files.
In embodiments of the present invention, if the quantity of documents of read data file is greater than configured data assembly tool
The file of the parameter of SQLLoader loads limit value, then data file is split as to m sub- data files in sequence, wherein
The quantity of documents of first sub- data file to m-1 sub- data files is equal to the data assembly tool SQLLoader's
The file of parameter loads limit value, wherein and in the multiple subfiles split, the data volume of different subfiles can be equal,
It can not also wait, it is not limited in the embodiment of the present invention；In the present embodiment, using operating system command to data file into
Row is split, and the fractionation order used for different operating system is not identical, for example, the present embodiment is with LINUX operating system
Example is illustrated, for LINUX operating system using SPLIT order by a file declustering at several subfiles, fractionation
Mode is included the following: 1, is split with line number, the format of order are as follows: " file after SPLIT-1 line number size original document is split
Name prefix " indicates to split file original document with fixed line number；2, it is split with size, the format of order
Are as follows: " SPLIT-b file size original document split after filename prefix ", that is, indicate by original document with glue file size into
Row is split.
Specifically, as shown in figure 5, the step S106 includes the following steps S401~S402:
The data file is split as the m subdata files according to document order, wherein first by step S401
A sub- data file is equal to the parameter of the data assembly tool SQLLoader to the quantity of documents of m-1 sub- data files
File load limit value.
In embodiments of the present invention, the data file is split as the m subdata files according to document order,
In, the value of m be limit value is loaded according to the file of the parameter of the data assembly tool SQLLoader to determine, that is,
It says, when the file of parameter of the quantity of documents of read data file greater than configured data assembly tool SQLLoader
Limit value is loaded, data file is split in sequence with the file load limit value of the parameter of data assembly tool SQLLoader,
Wherein, the quantity of documents of first sub- data file to m-1 sub- data files is equal to the data assembly tool
The file of the parameter of SQLLoader loads limit value.Specifically, selecting data text when splitting to the data file
First data in part are as fractionation starting point.
Step S402 deletes the last subdata file for dividing part.
Step S107 starts multiple sons of the data assembly tool SQLLoader according to the multiple subdata file
Process carries out loaded in parallel to the multiple subdata file, and the multiple subdata file is loaded onto the tables of data.
In embodiments of the present invention, after the data file is split as multiple subdata files, for each subdata text
Part creates a corresponding subprocess and multiple subdata files is concurrently loaded onto institute further according to multiple subprocess of creation
It states in tables of data.
Compared with prior art, a kind of data file loading method of the embodiment of the present invention has the following beneficial effects:
A kind of data file loading method of the embodiment of the present invention needs to import the number of the specified table structure of data by creation
According to table；Read control file and data file；Obtain the parameter configured for data assembly tool SQLLoader；Judge institute
Whether the data format for stating data file matches with the table structure of the tables of data created；If the data lattice of the data file
The table structure of formula and the tables of data created matches, according to the parameter of the data assembly tool SQLLoader configured, judgement
The file whether quantity of documents of read data file is greater than the parameter of configured data assembly tool SQLLoader adds
Carry limit value；If the quantity of documents of read data file is greater than the parameter of configured data assembly tool SQLLoader
File load limit value, according to the file of the parameter of the data assembly tool SQLLoader load limit value to the number
It is split according to file, obtains multiple subdata files；According to the multiple subdata file, start the data assembly tool
Multiple subprocess of SQLLoader carry out loaded in parallel to the multiple subdata file, by the multiple subdata file
It is loaded onto the tables of data.It can be realized the loaded in parallel of data file using the present invention, and improve the efficiency of file load.
Referring to Fig. 6, a kind of corresponding above-mentioned data file loading method, the embodiment of the present invention also propose a kind of data file
Loading device, as shown in fig. 6, the data file loading device 100 includes creating unit 101, reading unit 102, configuration unit
103, the first judging unit 104, second judgment unit 105, split cells 106, start-up loading unit 107, wherein
Creating unit 101 needs to import the tables of data of the specified table structure of data for creating；
Reading unit 102, for reading control file and data file；
Acquiring unit 103, for obtaining the parameter for being directed to data assembly tool SQLLoader and being configured；
First judging unit 104, for judge the data file data format whether with the tables of data that is created
Table structure matches；
Second judgment unit 105, if the table structure of data format and the tables of data created for the data file
Match, according to the parameter of the data assembly tool SQLLoader configured, judges the quantity of documents of read data file
Whether the file of the parameter greater than configured data assembly tool SQLLoader loads limit value；
Split cells 106, if the quantity of documents for read data file is greater than configured data assembly tool
The file of the parameter of SQLLoader loads limit value, is loaded according to the file of the parameter of the data assembly tool SQLLoader
Limit value splits the data file, obtains multiple subdata files；
Start-up loading unit 107, for starting the data assembly tool according to the multiple subdata file
Multiple subprocess of SQLLoader carry out loaded in parallel to the multiple subdata file, by the multiple subdata file
It is loaded onto the tables of data.
In some embodiments, as shown in fig. 7, the creating unit 101, comprising:
First determination unit 101a, the table name of the tables of data for determining specified table structure to be created, column name,
The maximum number of digits of data type and data type；
Subelement 101b is created, for the maximum according to identified table name, column name, data type and data type
Digit creates the tables of data of the specified table structure using the CREATE TABLE sentence of sql like language.
In some embodiments, as shown in figure 8, the acquiring unit 103, comprising:
Subelement 103a is obtained, for obtaining ROWS parameter, BINDSIZE and the ginseng of data assembly tool SQLLoader
Number READ BUFFER parameter, wherein the record number read every time is necessarily less than the value of ROWS parameter, and size must not exceed
The value of BINDSIZE parameter.
In some embodiments, as shown in figure 9, the judging unit 104, comprising:
Analytical unit 104a, for analyzing the data format of data in the data file；
Second determination unit 104b, for determining the data class of data according to the data format of data in the data file
Type and data type digit；
Judging unit 104c, if the data type phase of data type and the tables of data created for identified data
The data type digit of corresponding and identified data is corresponding with the maximum number of digits of the tables of data created, determines the data
The data format of file and the table structure of the tables of data created match.
In some embodiments, as shown in Figure 10, the split cells 106, comprising:
Subelement 106a is split, for the data file to be split as m institute according to document order and pre-set space
State subdata file, wherein the quantity of documents of first sub- data file to m-1 sub- data files is filled equal to the data
The file of parameter with tool SQLLoader loads limit value；
Unit 106b is deleted, for deleting the last subdata file for dividing part.
As seen from the above, the embodiment of the present invention needs to import the tables of data of the specified table structure of data by creation；It reads
Control file and data file；Obtain the parameter configured for data assembly tool SQLLoader；Judge the data file
Data format whether match with the table structure of the tables of data created；If the data format of the data file with created
The table structure of tables of data match, according to the parameter of the data assembly tool SQLLoader configured, judge read number
Whether it is greater than the file load limit value of the parameter of configured data assembly tool SQLLoader according to the quantity of documents of file；
If the file load of parameter of the quantity of documents of read data file greater than configured data assembly tool SQLLoader
Limit value loads limit value according to the file of the parameter of the data assembly tool SQLLoader and carries out to the data file
It splits, obtains multiple subdata files；According to the multiple subdata file, start the data assembly tool SQLLoader
Multiple subprocess loaded in parallel is carried out to the multiple subdata file, the multiple subdata file is loaded onto described
Tables of data.It can be realized the loaded in parallel of data file using the present invention, and improve the efficiency of file load.
Above-mentioned data file loading device and recording file format conversion the above method correspond, specific principle and
Process is identical as above-described embodiment the method, repeats no more.
Above-mentioned data file loading device can be implemented as a kind of form of computer program, and computer program can be such as
It is run in computer equipment shown in Figure 11.
Figure 11 is a kind of structure composition schematic diagram of computer equipment of the present invention.The equipment can be terminal, be also possible to
Server, wherein terminal can be smart phone, tablet computer, laptop, desktop computer, personal digital assistant and wear
Wear the electronic device that formula device etc. has communication function and speech voice input function.Server can be independent server, can also
To be server cluster that multiple servers form.Referring to Fig.1 1, which includes being connected by system bus 501
Processor 502, non-volatile memory medium 503, built-in storage 504 and the network interface 505 connect.Wherein, the computer equipment
500 non-volatile memory medium 503 can storage program area 5031 and computer program 5032,5032 quilt of computer program
When execution, processor 502 may make to execute a kind of data file loading method.The processor 502 of the computer equipment 500 is used for
Calculating and control ability are provided, the operation of entire computer equipment 500 is supported.The built-in storage 504 is non-volatile memories Jie
The operation of computer program 5032 in matter 503, which provides environment, may make processor when the computer program is executed by processor
502 execute a kind of data file loading method.The network interface 505 of computer equipment 500 is for carrying out network communication.This field
Technical staff is appreciated that structure shown in Figure 11, only the block diagram of part-structure relevant to application scheme, not
The restriction for the computer equipment being applied thereon to application scheme is constituted, specific computer equipment may include than in figure
Shown more or fewer components perhaps combine certain components or with different component layouts.
Wherein, following operation is realized when the processor 502 executes the computer program:
Creation needs to import the tables of data of the specified table structure of data；
Read control file and data file；
Obtain the parameter configured for data assembly tool SQLLoader；
Judge whether the data format of the data file matches with the table structure of the tables of data created；
If the data format of the data file and the table structure of the tables of data created match, according to the number configured
According to the parameter of assembly tool SQLLoader, judge whether the quantity of documents of read data file is greater than configured data
The file of the parameter of assembly tool SQLLoader loads limit value；
If parameter of the quantity of documents of read data file greater than configured data assembly tool SQLLoader
File loads limit value, loads limit value to the data according to the file of the parameter of the data assembly tool SQLLoader
File is split, and multiple subdata files are obtained；
According to the multiple subdata file, start multiple subprocess of the data assembly tool SQLLoader to institute
It states multiple subdata files and carries out loaded in parallel, the multiple subdata file is loaded onto the tables of data.
It is in one embodiment, described to create the tables of data for needing to import the specified table structure of data, comprising:
Determine the table name of the tables of data of specified table structure to be created, column name, data type and data type
Maximum number of digits；
According to the maximum number of digits of identified table name, column name, data type and data type, sql like language is used
CREATE TABLE sentence creates the tables of data of the specified table structure.
It is in one embodiment, described to obtain the parameter configured for data assembly tool SQLLoader, comprising:
ROWS parameter, BINDSIZE and the parameter READ BUFFER parameter of data assembly tool SQLLoader are obtained,
Wherein, the record number read every time is necessarily less than the value of ROWS parameter, and size must not exceed the value of BINDSIZE parameter.
In one embodiment, the data format for judging the data file whether the table with the tables of data created
Structure matches, comprising:
Analyze the data format of data in the data file；
The data type and data type digit of data are determined according to the data format of data in the data file；
If the data type of identified data number corresponding and identified with the data type of the tables of data created
According to data type digit it is corresponding with the maximum number of digits of the tables of data created, determine the data format of the data file with
The table structure of the tables of data created matches.
In one embodiment, the file of the parameter according to the data assembly tool SQLLoader loads limit
Value splits the data file, obtains multiple subdata files, comprising:
The data file is split as the m subdata files according to document order and pre-set space, wherein the
One sub- data file is equal to the ginseng of the data assembly tool SQLLoader to the quantity of documents of m-1 sub- data files
Several files loads limit value；
Delete the last subdata file for dividing part.
It will be understood by those skilled in the art that the embodiment of computer equipment shown in Figure 11 is not constituted to computer
The restriction of equipment specific composition, in other embodiments, computer equipment may include components more more or fewer than diagram, or
Person combines certain components or different component layouts.For example, in some embodiments, computer equipment only includes memory
And processor, in such embodiments, the structure and function of memory and processor are consistent with embodiment illustrated in fig. 11, herein
It repeats no more.
The present invention provides a kind of computer readable storage medium, computer-readable recording medium storage has one or one
A above computer program, the one or more computer program can be held by one or more than one processor
Row, to perform the steps of
Creation needs to import the tables of data of the specified table structure of data；
Read control file and data file；
Obtain the parameter configured for data assembly tool SQLLoader；
Judge whether the data format of the data file matches with the table structure of the tables of data created；
If the data format of the data file and the table structure of the tables of data created match, according to the number configured
According to the parameter of assembly tool SQLLoader, judge whether the quantity of documents of read data file is greater than configured data
The file of the parameter of assembly tool SQLLoader loads limit value；
If parameter of the quantity of documents of read data file greater than configured data assembly tool SQLLoader
File loads limit value, loads limit value to the data according to the file of the parameter of the data assembly tool SQLLoader
File is split, and multiple subdata files are obtained；
According to the multiple subdata file, start multiple subprocess of the data assembly tool SQLLoader to institute
It states multiple subdata files and carries out loaded in parallel, the multiple subdata file is loaded onto the tables of data.
It is in one embodiment, described to create the tables of data for needing to import the specified table structure of data, comprising:
Determine the table name of the tables of data of specified table structure to be created, column name, data type and data type
Maximum number of digits；
According to the maximum number of digits of identified table name, column name, data type and data type, sql like language is used
CREATE TABLE sentence creates the tables of data of the specified table structure.
It is in one embodiment, described to obtain the parameter configured for data assembly tool SQLLoader, comprising:
ROWS parameter, BINDSIZE and the parameter READ BUFFER parameter of data assembly tool SQLLoader are obtained,
Wherein, the record number read every time is necessarily less than the value of ROWS parameter, and size must not exceed the value of BINDSIZE parameter.
In one embodiment, the data format for judging the data file whether the table with the tables of data created
Structure matches, comprising:
Analyze the data format of data in the data file；
The data type and data type digit of data are determined according to the data format of data in the data file；
If the data type of identified data number corresponding and identified with the data type of the tables of data created
According to data type digit it is corresponding with the maximum number of digits of the tables of data created, determine the data format of the data file with
The table structure of the tables of data created matches.
In one embodiment, the file of the parameter according to the data assembly tool SQLLoader loads limit
Value splits the data file, obtains multiple subdata files, comprising:
The data file is split as the m subdata files according to document order and pre-set space, wherein the
One sub- data file is equal to the ginseng of the data assembly tool SQLLoader to the quantity of documents of m-1 sub- data files
Several files loads limit value；
Delete the last subdata file for dividing part.
Present invention storage medium above-mentioned include: magnetic disk, CD, read-only memory (Read-Only Memory,
The various media that can store program code such as ROM).
Unit in all embodiments of the invention can pass through universal integrated circuit, such as CPU (Central
Processing Unit, central processing unit), or pass through ASIC (Application Specific Integrated
Circuit, specific integrated circuit) Lai Shixian.
Step in data file loading method of the embodiment of the present invention can the adjustment of carry out sequence, merging according to actual needs
With delete.
Unit in data file loading device of the embodiment of the present invention can be merged according to actual needs, divides and be deleted
Subtract.
The above description is merely a specific embodiment, but scope of protection of the present invention is not limited thereto, any
Those familiar with the art in the technical scope disclosed by the present invention, can readily occur in various equivalent modifications or replace
It changes, these modifications or substitutions should be covered by the protection scope of the present invention.Therefore, protection scope of the present invention should be with right
It is required that protection scope subject to.
Claims (10)
1. a kind of data file loading method, which is characterized in that described method includes following steps:
Creation needs to import the tables of data of the specified table structure of data；
Read control file and data file；
Obtain the parameter configured for data assembly tool SQLLoader；
Judge whether the data format of the data file matches with the table structure of the tables of data created；
If the data format of the data file and the table structure of the tables of data created match, filled according to the data configured
Parameter with tool SQLLoader, judges whether the quantity of documents of read data file is greater than configured data assembly
The file of the parameter of tool SQLLoader loads limit value；
If the file of parameter of the quantity of documents of read data file greater than configured data assembly tool SQLLoader
Limit value is loaded, limit value is loaded to the data file according to the file of the parameter of the data assembly tool SQLLoader
It is split, obtains multiple subdata files；
According to the multiple subdata file, start multiple subprocess of the data assembly tool SQLLoader to described more
A sub- data file carries out loaded in parallel, and the multiple subdata file is loaded onto the tables of data.
2. data file loading method as described in claim 1, which is characterized in that described create needs to import the specified of data
The tables of data of table structure, comprising:
Determine the table name, column name, the maximum of data type and data type of the tables of data of specified table structure to be created
Digit；
According to the maximum number of digits of identified table name, column name, data type and data type, sql like language is used
CREATE TABLE sentence creates the tables of data of the specified table structure.
3. data file loading method as described in claim 1, which is characterized in that described obtain is directed to data assembly tool
The parameter that SQLLoader is configured, comprising:
Obtain ROWS parameter, BINDSIZE and the parameter READ BUFFER parameter of data assembly tool SQLLoader, wherein
The record number read every time is necessarily less than the value of ROWS parameter, and size must not exceed the value of BINDSIZE parameter.
4. data file loading method as described in claim 1, which is characterized in that the data of the judgement data file
Whether format matches with the table structure of the tables of data created, comprising:
Analyze the data format of data in the data file；
The data type and data type digit of data are determined according to the data format of data in the data file；
If the data type of identified data data corresponding and identified with the data type of the tables of data created
Data type digit is corresponding with the maximum number of digits of the tables of data created, determines the data format of the data file and is created
The table structure for the tables of data built matches.
5. data file loading method as described in claim 1, which is characterized in that described according to the data assembly tool
The file load limit value of the parameter of SQLLoader splits the data file, obtains multiple subdata files, wraps
It includes:
The data file is split as the m subdata files according to document order, wherein first sub- data file arrives
The file of parameter of the quantity of documents of m-1 sub- data files equal to the data assembly tool SQLLoader loads limit
Value；
Delete the last subdata file for dividing part.
6. a kind of data file loading device, which is characterized in that the data file loading device includes:
Creating unit needs to import the tables of data of the specified table structure of data for creating；
Reading unit, for reading control file and data file；
Acquiring unit, for obtaining the parameter for being directed to data assembly tool SQLLoader and being configured；
First judging unit, for judge the data file data format whether the table structure phase with the tables of data created
Matching；
Second judgment unit, if the table structure of data format and the tables of data created for the data file matches,
According to the parameter of the data assembly tool SQLLoader configured, judge whether the quantity of documents of read data file is big
Limit value is loaded in the file of the parameter of the data assembly tool SQLLoader configured；
Split cells, if the quantity of documents for read data file is greater than configured data assembly tool
The file of the parameter of SQLLoader loads limit value, is loaded according to the file of the parameter of the data assembly tool SQLLoader
Limit value splits the data file, obtains multiple subdata files；
Start-up loading unit, for starting the more of the data assembly tool SQLLoader according to the multiple subdata file
A subprocess carries out loaded in parallel to the multiple subdata file, and the multiple subdata file is loaded onto the data
Table.
7. data file loading device as claimed in claim 6, which is characterized in that the creating unit, comprising:
First determination unit, table name, column name, the data type of the tables of data for determining specified table structure to be created
With the maximum number of digits of data type；
Creation subelement makes for the maximum number of digits according to identified table name, column name, data type and data type
The tables of data of the specified table structure is created with the CREATE TABLE sentence of sql like language.
8. data file loading device as claimed in claim 6, which is characterized in that the acquiring unit, comprising:
Subelement is obtained, for obtaining ROWS parameter, BINDSIZE and the parameter READ of data assembly tool SQLLoader
BUFFER parameter, wherein the record number read every time is necessarily less than the value of ROWS parameter, and size must not exceed BINDSIZE
The value of parameter.
9. a kind of computer equipment, including memory, processor and it is stored on the memory and can be on the processor
The computer program of operation, which is characterized in that the processor realizes that claim 1-5 such as appoints when executing the computer program
Data file loading method described in one.
10. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage have one or
More than one computer program, the one or more computer program can be by one or more than one processors
It executes, to realize data file loading method as described in any one in claim 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910462330.5A CN110347673A (en) | 2019-05-30 | 2019-05-30 | Data file loading method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910462330.5A CN110347673A (en) | 2019-05-30 | 2019-05-30 | Data file loading method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110347673A true CN110347673A (en) | 2019-10-18 |
Family
ID=68174457
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910462330.5A Pending CN110347673A (en) | 2019-05-30 | 2019-05-30 | Data file loading method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110347673A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125769A (en) * | 2019-12-27 | 2020-05-08 | 上海轻维软件有限公司 | Mass data desensitization method based on ORACLE database |
| CN111597244A (en) * | 2020-05-19 | 2020-08-28 | 北京思特奇信息技术股份有限公司 | Method and system for quickly importing data and computer storage medium |
| CN112001160A (en) * | 2020-08-27 | 2020-11-27 | 中国平安财产保险股份有限公司 | Data processing method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103077241A (en) * | 2013-01-10 | 2013-05-01 | 中国银行股份有限公司 | Method for loading data in parallel after splitting files |
| US20140114924A1 (en) * | 2012-10-19 | 2014-04-24 | International Business Machines Corporation | Data loading tool |
| CN105869048A (en) * | 2016-03-28 | 2016-08-17 | 中国建设银行股份有限公司 | Data processing method and system |
| CN106934037A (en) * | 2017-03-15 | 2017-07-07 | 郑州云海信息技术有限公司 | A kind of high concurrent realizes the method that database quickly loads data |
| CN109726244A (en) * | 2019-01-29 | 2019-05-07 | 北京中电普华信息技术有限公司 | Data lead-in method and device |
-
2019
- 2019-05-30 CN CN201910462330.5A patent/CN110347673A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140114924A1 (en) * | 2012-10-19 | 2014-04-24 | International Business Machines Corporation | Data loading tool |
| CN103077241A (en) * | 2013-01-10 | 2013-05-01 | 中国银行股份有限公司 | Method for loading data in parallel after splitting files |
| CN105869048A (en) * | 2016-03-28 | 2016-08-17 | 中国建设银行股份有限公司 | Data processing method and system |
| CN106934037A (en) * | 2017-03-15 | 2017-07-07 | 郑州云海信息技术有限公司 | A kind of high concurrent realizes the method that database quickly loads data |
| CN109726244A (en) * | 2019-01-29 | 2019-05-07 | 北京中电普华信息技术有限公司 | Data lead-in method and device |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111125769A (en) * | 2019-12-27 | 2020-05-08 | 上海轻维软件有限公司 | Mass data desensitization method based on ORACLE database |
| CN111125769B (en) * | 2019-12-27 | 2023-09-19 | 上海轻维软件有限公司 | Mass data desensitization method based on ORACLE database |
| CN111597244A (en) * | 2020-05-19 | 2020-08-28 | 北京思特奇信息技术股份有限公司 | Method and system for quickly importing data and computer storage medium |
| CN112001160A (en) * | 2020-08-27 | 2020-11-27 | 中国平安财产保险股份有限公司 | Data processing method, device, equipment and storage medium |
| CN112001160B (en) * | 2020-08-27 | 2023-07-28 | 中国平安财产保险股份有限公司 | Data processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111241389B (en) | Sensitive word filtering method and device based on matrix, electronic equipment and storage medium | |
| CN108491475A (en) | Data rapid batch introduction method, electronic device and computer readable storage medium | |
| US20060294049A1 (en) | Back-off mechanism for search | |
| CN110347673A (en) | Data file loading method, device, computer equipment and storage medium | |
| CN112560100B (en) | Data desensitizing method and device, computer readable storage medium and electronic equipment | |
| CN106547911B (en) | Access method and system for massive small files | |
| CN109542907A (en) | Database caches construction method, device, computer equipment and storage medium | |
| US20100251227A1 (en) | Binary resource format and compiler | |
| CN110020358B (en) | Method and device for generating dynamic page | |
| CN109766085A (en) | A kind of method and device handling enumeration type code | |
| CN106528896A (en) | Database optimization method and apparatus | |
| CN113672204A (en) | Interface document generation method, system, electronic equipment and storage medium | |
| CN106648569A (en) | Target serialization achieving method and device | |
| CN109960554A (en) | Show method, equipment and the computer storage medium of reading content | |
| JP5699743B2 (en) | SEARCH METHOD, SEARCH DEVICE, AND COMPUTER PROGRAM | |
| CN106202220A (en) | The method of data and device in a kind of reading object storage system | |
| CN110502506A (en) | A kind of data processing method, device, equipment and storage medium | |
| CN110580212B (en) | Data export method and device of application program, electronic equipment and storage medium | |
| US8595095B2 (en) | Framework for integrated storage of banking application data | |
| CN112162982A (en) | Data query method, device, equipment and medium | |
| US20130226619A1 (en) | Input support device and input support method | |
| CN110795920A (en) | Document generation method and device | |
| CN114968917A (en) | Method and device for rapidly importing file data | |
| CN100397399C (en) | Method and device for supporting multi-languages in FAT file system | |
| CN112632266B (en) | Data writing method and device, computer equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191018 |