CN109284260A - Large data files read method, device, computer equipment and storage medium - Google Patents
Large data files read method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN109284260A CN109284260A CN201811203112.1A CN201811203112A CN109284260A CN 109284260 A CN109284260 A CN 109284260A CN 201811203112 A CN201811203112 A CN 201811203112A CN 109284260 A CN109284260 A CN 109284260A
- Authority
- CN
- China
- Prior art keywords
- data
- field
- target
- field data
- dbf file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
- G06F12/023—Free address space management
- G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
- G06F12/0246—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory in block erasable memory, e.g. flash memory
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of large data files read method, device, computer equipment and storage mediums, are applied to database technical field, for solving the problems, such as the data reading mode inefficiency of existing dbf file.The method include that reading the header information of target dbf file, every row field data amount in header information is obtained；First piece of field data of target dbf file is mapped in specified memory as present field block；Successively obtain the field data of every row in present field block line by line according to every row field data amount；The field data got is parsed, the data value after being parsed；Until before getting all field datas of target dbf file, next piece of field data of target dbf file is mapped in specified memory as new present field block, returns and executes the step of successively obtaining the field data of every row in present field block line by line according to every row field data amount.
Description
Technical field
The present invention relates to database technical field more particularly to large data files read method, device, computer equipment and
Storage medium.
Background technique
With the arrival of big data era, storage, reading and the migration of data are become more and more important.Currently, database
Dbf formatted file is mostly used to carry out storing data greatly.Dbf file (database file of the entitled .dbf of suffix) is dBase, FoxPro
Etc. database format file used in Database Systems.Why big more options use dbf file, this is because when big data
Very huge for required data volume, server generally requires to get required data simultaneously from data center or other places
It stores in database, it can be with quick response when so as to user end to server request data.Dbf file is because it is with good
Data structure and be widely used.
When reading dbf file, existing way is carried out according to the requirement of dbf field format to every data from disk
It is successively read, and parses the field data in dbf file in reading process, obtain the value of respective field.
However, the existing method read big data quantity dbf file when, such as read 10G or more dbf file, it is past
Toward needing to expend a large amount of read access time, it is difficult to meet big data era to data the needs of quickly reading.
Summary of the invention
The embodiment of the present invention provides a kind of large data files read method, device, computer equipment and storage medium, with solution
The problem of data reading mode inefficiency of certainly existing dbf file.
A kind of large data files read method, comprising:
The header information for reading target dbf file, obtains every row field data amount in the header information, every row
Field data amount refers to the size of every a line field data in the target dbf file；
First piece of field data of the target dbf file is mapped in specified memory as present field block, the head
Block field data refers to the preset data for being located at after the header information and abutting the header information in the target dbf file
Measure the field data of size；
Successively obtain the field data of every row in the present field block line by line according to every row field data amount；
The field data got is parsed, the data value after being parsed；
Until before getting all field datas of the target dbf file, by next piece of the target dbf file
Field data maps in the specified memory as new present field block, returns and executes according to every row field data amount
The step of successively obtaining the field data of every row in the present field block line by line, the next piece of field data refer to the mesh
Mark the field data for being located at the preset data amount size after present field block and against present field block in dbf file.
A kind of large data files reading device, comprising:
Header information read module obtains every in the header information for reading the header information of target dbf file
Row field data amount, every row field data amount refer to the size of every a line field data in the target dbf file；
Field data mapping block, for mapping to first piece of field data of the target dbf file in specified memory
As present field block, the first piece of field data, which refers to, to be located at after the header information and is abutted in the target dbf file
The field data of the preset data amount size of the header information；
Module is obtained line by line, it is every in the present field block for successively being obtained line by line according to every row field data amount
Capable field data；
Field data parsing module, for being parsed to the field data got, the data value after being parsed；
Circular treatment module, before all field datas up to getting the target dbf file, by the mesh
Next piece of field data of mark dbf file is mapped to as new present field block in the specified memory, and return successively triggers
Acquisition module and the field data parsing module, the next piece of field data line by line refer to the target dbf file
In be located at present field block after and against present field block the preset data amount size field data.
A kind of computer equipment, including memory, processor and storage are in the memory and can be in the processing
The computer program run on device, the processor realize above-mentioned large data files read method when executing the computer program
The step of.
A kind of computer readable storage medium, the computer-readable recording medium storage have computer program, the meter
The step of calculation machine program realizes above-mentioned large data files read method when being executed by processor.
Above-mentioned large data files read method, device, computer equipment and storage medium, firstly, reading target dbf file
Header information, obtain every row field data amount in the header information, every row field data amount refers to the target
The size of every a line field data in dbf file；Then, first piece of field data of the target dbf file is mapped to specified
Present field block is used as in memory, the first piece of field data refers in the target dbf file after the header information
And the field data of the preset data amount size against the header information；Then, successively according to every row field data amount
The field data of every row in the present field block is obtained line by line；The field data got is parsed, after obtaining parsing
Data value；Until before getting all field datas of the target dbf file, by the next of the target dbf file
Block field data maps in the specified memory as new present field block, returns and executes according to every row field data
The step of amount successively obtains the field data of every row in the present field block line by line, the next piece of field data refer to described
It is located at the field data of the preset data amount size after present field block and against present field block in target dbf file.
As it can be seen that the speed due to handling data in memory will be added far faster than the speed for handling data in disk in the present invention
The mode of File Mapping is also higher than reading the efficiency of data on disk, therefore, is higher than on whole treatment effeciency existing
Mode improves the efficiency that dbf file reads data, reduces the read access time of data, can satisfy big data era to data
The demand quickly read.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention
Example, for those of ordinary skill in the art, without any creative labor, can also be according to these attached drawings
Obtain other attached drawings.
Fig. 1 is an application environment schematic diagram of large data files read method in one embodiment of the invention；
Fig. 2 is a flow chart of large data files read method in one embodiment of the invention；
Fig. 3 is process of the large data files read method step 102 under an application scenarios in one embodiment of the invention
Schematic diagram；
Fig. 4 is that large data files read method deletes header information under an application scenarios in one embodiment of the invention
Flow diagram；
Fig. 5 is that this is preset under an application scenarios is default for large data files read method in one embodiment of the invention
The flow diagram of data volume size；
Fig. 6 is the structural schematic diagram of large data files reading device in one embodiment of the invention；
Fig. 7 is a schematic diagram of computer equipment in one embodiment of the invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
Large data files read method provided by the present application, can be applicable in the application environment such as Fig. 1, wherein client
It is communicated by network with server.Wherein, the client can be, but not limited to various personal computers, laptop,
Smart phone, tablet computer and portable wearable device.Server can use independent server either multiple servers
The server cluster of composition is realized.
In one embodiment, it as shown in Fig. 2, providing a kind of large data files read method, applies in Fig. 1 in this way
Server for be illustrated, include the following steps:
101, the header information for reading target dbf file, obtains every row field data amount in the header information, described
Every row field data amount refers to the size of every a line field data in the target dbf file；
In the present embodiment, dbf file is one kind of database format file, there is fixed call format, such as dbf text
The head of part is header information, and the size of a line field data in the dbf file, i.e., described every row word are had recorded in header information
Segment data amount.Therefore, server can be every in the header information to obtain by reading the header information of target dbf file
Row field data amount.
102, first piece of field data of the target dbf file is mapped in specified memory as present field block, institute
First piece of field data is stated to refer in the target dbf file after the header information and against the default of the header information
The field data of data volume size；
It is understood that due to map the data into memory rate will far faster than read data rate, this
Target dbf file is mapped in specified memory by the way of File Mapping and is handled by scheme.Since target dbf file is logical
It is often larger, and the memory source of server is limited, and is generally difficult to accomplish disposably to map to entire target dbf file interior
In depositing.Therefore, in the present embodiment, server can first be mapped to first piece of field data of the target dbf file in specified
It deposits middle as present field block, wherein the first piece of field data, which refers to, is located at the header information in the target dbf file
The field data of preset data amount size afterwards and against the header information.It illustrates, it is assumed that 0- in the target dbf file
100 bytes are header information, and 101-10000 byte is field data.When the preset data amount size is 100 byte, then should
First piece of field data is 101-200 byte, and server first maps the field data of 101-200 byte in the target dbf file
Present field block is used as into the specified memory.
In actual use, when target dbf file is mapped to specified memory, if thinking the first piece of field data individually
Specified memory is mapped to, this is difficult to realize.For this purpose, as shown in figure 3, further, the step 102 specifically can wrap
It includes:
201, the data for presetting mapping data volume size in the target dbf file since the header information are mapped
To specified memory, wherein the default mapping data volume size is equal to the data of preset data amount size and the header information
Measure the sum of size；
202, the data in the data for mapping to the specified memory in addition to the header information are determined as current word
Section block.
For step 201, it is to be understood that in order to which first piece of field data is successfully mapped to specified memory, can incite somebody to action
The first piece of field data maps to specified memory together with header information, namely maps since the head of target dbf file
The field data of default mapping data volume size, wherein the default mapping data volume size is equal to preset data amount size and institute
State the sum of the data volume size of header information.For example, accept the example above, i.e., it will be to the number of 0-200 byte in target dbf file
According to mapping in specified memory.
For step 202, since the data mapped in step 201 contain header information, and do not have in these header informations
There is the value comprising field data needed for this programme, therefore, server will can directly map to the data of the specified memory
In data in addition to the header information be determined as present field block.It, can for the header information being mapped in specified memory
To wipe or ignore, the present embodiment is not construed as limiting this.
Aiming at the problem that being difficult to realize individually map first piece of field data when mapping data mentioned above, in the present embodiment
There are also another ways to be handled.As shown in figure 4, further, before step 102 further include:
301, the header information of the target dbf file is deleted, obtains new target dbf file；
Step 102 specifically: reflect the field data for being located at the preset data amount size on head in the target dbf file
It is incident upon in specified memory as present field block.
The thinking of this mode is, before mapping target dbf file, first by the head in target dbf file
Information deletion, in this way, being only left Field Count in new target dbf file when executing step 102 to the mapping of target dbf file
According to there is no header information brings to interfere, therefore can directly carry out mapping processing.
For step 301, server can be deleted the header information of the target dbf file, obtain deleting head letter
New target dbf file after breath.
On the basis of step 301, it is to be understood that in above-mentioned steps 102, server can be by the target dbf
Field data in file positioned at the preset data amount size on head maps in specified memory as present field block.This be because
Without header information in the target dbf file, therefore can directly to take from the beginning pre- by the processing of step 301
If the field data of data volume size is mapped in specified memory, it is as current that these are mapped to the field data in specified memory
Field block.
In dbf file, field data stores in rows, and the data volume of every row is every row field data
Amount.But, it is contemplated that when determining to map obtained present field block, the data volume of present field block is that preset data amount is big
Small, if being not limited to the preset data amount size and every row field data amount, this meeting is so that last in present field block
Capable data may insufficient a line field data.Such as, it is assumed that every row field data amount is 100, and preset data amount size is
250,0-100 bytes are header information, then first piece of field data of the target dbf file is 101-350, and last line data are
301-350, it is seen that last line data deficiencies a line field data, this can bring difficulty to subsequent parsing field data.For this purpose,
The present embodiment can avoid the occurrence of above situation by limiting the relationship of preset data amount size and every row field data amount, make
The last line data for obtaining each present field block are enough a line field datas.
As shown in figure 5, further, the preset data amount size can be preset by following steps:
401, the current space available of the specified memory is obtained；
402, determine the specified memory for mapping according to preset memory use ratio and the current space available
The mapping space of field data；
403, the mapping space is obtained into the first numerical value divided by every row field data amount；
404, be rounded first numerical value, and calculate first numerical value after being rounded and every row field data amount it
Product, obtains second value as preset data amount size.
For step 401, which refers to the remaining space that specified memory currently can be used, for example, false
If specified memory tribute 4g, has used 2g, then current space available is 2g.
For step 402, it is to be understood that server can preset memory use ratio, the memory use than
Example indicates that data should be mapped using the memory of how much ratios in current space available.The memory use ratio can basis
Situation setting is actually used, for example can be set as 50%.Server is in the current space available for getting the specified memory
Later, the product that can calculate preset memory use ratio and the current space available obtains the specified memory for reflecting
Penetrate the mapping space of field data.
For step 403 and 404, in the present embodiment, in order to enable preset data amount size is equal to every row field data amount
Integral multiple, while the preset data amount size is as close possible to the size of the mapping space.It therefore, can be first by the mapping
Space obtains the first numerical value divided by every row field data amount, is then rounded to the first numerical value, first numerical value after rounding
It may be considered multiple, the second number that the product of first numerical value after calculating the rounding and every row field data amount obtains
Value is the integral multiple of every row field data amount, may thereby determine that the second value is preset data amount size.
103, the field data of every row in the present field block is successively obtained line by line according to every row field data amount；
It is understood that due in dbf file field data be the arrangement of a line a line, storage, and the word of every a line
In segment data again made an appointment each field put in order and each field respectively shared by byte therefore obtaining
When field data in the present field block, need successively to obtain the present field line by line according to every row field data amount
The field data of every row in block, then executing step 104, these field datas are parsed respectively.
Further, as shown in the above, the possible insufficient a line field data of the last line data of present field block,
Faced with this situation, after step 103, if this method can also include: the present field block last line got
The data volume of field data be less than every row field data amount, then keep in the field data of the last line as connecting number
According to；Then, next piece of field data of the target dbf file is mapped in the specified memory as new in step 105
Present field block after, return execute successively is obtained line by line in the present field block according to every row field data amount it is every
Before the step of capable field data, the subsequent data is incorporated into the head of the present field block.For example, accepting above-mentioned
Citing, it is assumed that present field block last line data are 301-350, then the last line data can be temporarily stored in interior by server
In depositing, then, after step 105 obtains new present field block, it is known that the new present field block is 351-600, service
The last line data are merged into the head of present field block by device, so that present field block is 301-600.As it can be seen that logical
The processing for crossing this mode can handle the last line data of present field block well.
104, the field data got is parsed, the data value after being parsed；
In the present embodiment, whenever getting in present field block after the field data of every row, server can be to obtaining
The field data got is parsed, the data value after being parsed.It is understood that in dbf file every a line field
Each field in data is according to the arrangement that puts in order made an appointment, and respectively shared byte is also preparatory for each field
It appoints, therefore, after getting these field datas, is directly parsed in these field datas according to each field of agreement
Data value.
It 105, will be under the target dbf file until before getting all field datas of the target dbf file
One piece of field data maps in the specified memory as new present field block, returns and executes according to every row Field Count
The step of successively obtaining the field data of every row in the present field block line by line according to amount, the next piece of field data refers to institute
State the Field Count for being located at the preset data amount size after present field block and against present field block in target dbf file
According to.
In the present embodiment, due to obtaining and parsing the field data in entire target dbf file, executing step
After 104, that is, after obtaining and having parsed present field block, server can be by next piece of Field Count of the target dbf file
According to mapping in the specified memory as new present field block, then return to step 103 and step 104 handle head
Next piece of field data of block field data after step 103 and step 104 processing complete next piece of field data, then determines
Under next piece of field data be new present field block, be then followed by acquisition and parsing, it is all until the target dbf file
Field data is acquired and is parsed, it is known that, the data of all field datas in the target dbf file available at this time
Value so far completes the reading of the target dbf file.
In the embodiment of the present invention, firstly, reading the header information of target dbf file, obtain every in the header information
Row field data amount, every row field data amount refer to the size of every a line field data in the target dbf file；So
Afterwards, first piece of field data of the target dbf file is mapped in specified memory as present field block, the first piece of field
Data refer to the preset data amount size for being located at after the header information and abutting the header information in the target dbf file
Field data；Then, the field of every row in the present field block is successively obtained line by line according to every row field data amount
Data；The field data got is parsed, the data value after being parsed；Until getting the target dbf file
All field datas before, next piece of field data of the target dbf file is mapped into conduct in the specified memory
New present field block returns to execution according to every row field data amount and successively obtains every row in the present field block line by line
Field data the step of, the next piece of field data refer to be located in the target dbf file it is after present field block and tight
By the field data of the preset data amount size of present field block.As it can be seen that in the present invention, due to handling data in memory
Speed will be far faster than the speed for handling data in disk, along with the mode of File Mapping is also than reading data on disk
Efficiency wants high, therefore, is higher than existing way on whole treatment effeciency, improves the efficiency that dbf file reads data, subtracts
The read access time of a small number of evidences can satisfy the demand that big data era quickly reads data.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
In one embodiment, a kind of large data files reading device is provided, the large data files reading device and above-mentioned reality
Large data files read method in example is applied to correspond.As shown in fig. 6, the large data files reading device includes that header information is read
Modulus block 501, obtains module 503, field data parsing module 504 and circular treatment mould at field data mapping block 502 line by line
Block 505.Detailed description are as follows for each functional module:
Header information read module 501 obtains in the header information for reading the header information of target dbf file
Every row field data amount, every row field data amount refers to the size of every a line field data in the target dbf file；
Field data mapping block 502, for first piece of field data of the target dbf file to be mapped to specified memory
Middle to be used as present field block, the first piece of field data refers to after the header information and tight in the target dbf file
By the field data of the preset data amount size of the header information；
Module 503 is obtained line by line, for successively obtaining the present field block line by line according to every row field data amount
In every row field data；
Field data parsing module 504, for being parsed to the field data got, the data after being parsed
Value；
Circular treatment module 505, before all field datas up to getting the target dbf file, by institute
Next piece of field data for stating target dbf file maps in the specified memory as new present field block, returns successively
Triggering acquisition module and the field data parsing module, the next piece of field data line by line refer to the target dbf
It is located at the field data of the preset data amount size after present field block and against present field block in file.
Further, the field data mapping block may include:
First map unit, for mapping data volume will to be preset in the target dbf file since the header information
The data of size map to specified memory, wherein the default mapping data volume size be equal to preset data amount size with it is described
The sum of data volume size of header information；
Field block determination unit, the number in the data for the specified memory will to be mapped in addition to the header information
According to being determined as present field block.
Further, the large data files reading device can also include:
Header information removing module obtains new target dbf for deleting the header information of the target dbf file
File；
The field data mapping block is specifically used for: will be located at the preset data amount on head in the target dbf file
The field data of size maps in specified memory as present field block.
Further, the preset data amount size can be preset by following steps:
Available space obtains module, for obtaining the current space available of the specified memory；
Mapping space determining module, for according to preset memory use ratio and current space available determination
Specified memory is used for the mapping space of map field data；
First Numerical Simulation Module, for the mapping space divided by every row field data amount, to be obtained the first number
Value；
Floor module for being rounded first numerical value, and calculates first numerical value and every row word after being rounded
The product of segment data amount obtains second value as preset data amount size.
Further, the large data files reading device can also include:
Temporary storage module, if the data volume of the field data for the present field block last line got is less than every
Row field data amount then keeps in the field data of the last line as subsequent data；
Merging module, for being mapped in the specified memory by next piece of field data of the target dbf file
After new present field block, return successively triggering it is described obtain line by line module and the field data parsing module it
Before, the subsequent data is incorporated into the head of the present field block.
Specific restriction about large data files reading device may refer to above for large data files read method
Restriction, details are not described herein.Modules in above-mentioned large data files reading device can be fully or partially through software, hard
Part and combinations thereof is realized.Above-mentioned each module can be embedded in the form of hardware or independently of in the processor in computer equipment,
It can also be stored in a software form in the memory in computer equipment, execute the above modules in order to which processor calls
Corresponding operation.
In one embodiment, a kind of computer equipment is provided, which can be server, internal junction
Composition can be as shown in Figure 7.The computer equipment include by system bus connect processor, memory, network interface and
Database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The memory packet of the computer equipment
Include non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer program and data
Library.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating
The database of machine equipment is for storing the data being related in large data files read method.The network interface of the computer equipment
For being communicated with external terminal by network connection.To realize a kind of big data text when the computer program is executed by processor
Part read method.
In one embodiment, a kind of computer equipment is provided, including memory, processor and storage are on a memory
And the computer program that can be run on a processor, processor realize big data text in above-described embodiment when executing computer program
The step of part read method, such as step 101 shown in Fig. 2 is to step 105.Alternatively, reality when processor executes computer program
The function of each module/unit of large data files reading device in existing above-described embodiment, such as module 501 shown in Fig. 6 is to module
505 function.To avoid repeating, which is not described herein again.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program realizes the step of large data files read method in above-described embodiment, such as step shown in Fig. 2 when being executed by processor
Rapid 101 to step 105.Alternatively, realizing that large data files read dress in above-described embodiment when computer program is executed by processor
The function for each module/unit set, such as module 501 shown in Fig. 6 is to the function of module 505.It is no longer superfluous here to avoid repeating
It states.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
To any reference of memory, storage, database or other media used in each embodiment provided herein,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing
The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (10)
1. a kind of large data files read method characterized by comprising
The header information for reading target dbf file, obtains every row field data amount in the header information, every row field
Data volume refers to the size of every a line field data in the target dbf file；
First piece of field data of the target dbf file is mapped in specified memory as present field block, the first piece of word
Segment data refers to that the preset data amount being located in the target dbf file after the header information and against the header information is big
Small field data；
Successively obtain the field data of every row in the present field block line by line according to every row field data amount；
The field data got is parsed, the data value after being parsed；
Until before getting all field datas of the target dbf file, by next piece of field of the target dbf file
Data map in the specified memory as new present field block, return and execute according to every row field data amount successively
The step of obtaining the field data of every row in the present field block line by line, the next piece of field data refer to the target
It is located at the field data of the preset data amount size after present field block and against present field block in dbf file.
2. large data files read method according to claim 1, which is characterized in that described by the target dbf file
First piece of field data map in specified memory and include: as present field block
The data for presetting mapping data volume size in the target dbf file since the header information are mapped in specified
Deposit, wherein the default mapping data volume size be equal to preset data amount size and the header information data volume size it
With；
Data in the data for mapping to the specified memory in addition to the header information are determined as present field block.
3. large data files read method according to claim 1, which is characterized in that by the target dbf file
First piece of field data maps in specified memory as before present field block, further includes:
The header information of the target dbf file is deleted, new target dbf file is obtained；
It is described to map to first piece of field data of the target dbf file in specified memory as present field block specifically:
The field data for being located at the preset data amount size on head in the target dbf file is mapped in specified memory as current
Field block.
4. large data files read method according to claim 1, which is characterized in that the preset data amount size passes through
Following steps are preset:
Obtain the current space available of the specified memory；
Determine the specified memory for map field data according to preset memory use ratio and the current space available
Mapping space；
By the mapping space divided by every row field data amount, the first numerical value is obtained；
It is rounded first numerical value, and calculates the product of first numerical value and every row field data amount after being rounded, is obtained
Second value is as preset data amount size.
5. large data files read method according to any one of claim 1 to 4, which is characterized in that according to described
Every row field data amount is successively obtained in the present field block line by line after the field data of every row, further includes:
If the data volume of the field data of the present field block last line got is less than every row field data amount, temporarily
The field data of the last line is deposited as subsequent data；
It maps in the specified memory by next piece of field data of the target dbf file as new present field block
Later, the field data for executing and successively obtaining every row in the present field block line by line according to every row field data amount is returned
The step of before, the subsequent data is incorporated into the head of the present field block.
6. a kind of large data files reading device characterized by comprising
Header information read module obtains every row word in the header information for reading the header information of target dbf file
Segment data amount, every row field data amount refer to the size of every a line field data in the target dbf file；
Field data mapping block, for first piece of field data of the target dbf file to be mapped to conduct in specified memory
Present field block, the first piece of field data refer in the target dbf file and are located at after the header information and against described
The field data of the preset data amount size of header information；
Module is obtained line by line, for successively obtaining every row in the present field block line by line according to every row field data amount
Field data；
Field data parsing module, for being parsed to the field data got, the data value after being parsed；
Circular treatment module, before all field datas up to getting the target dbf file, by the target
Next piece of field data of dbf file is mapped to as new present field block in the specified memory, and return successively triggers institute
Acquisition module and the field data parsing module, the next piece of field data line by line is stated to refer in the target dbf file
The field data of the preset data amount size after present field block and against present field block.
7. large data files reading device according to claim 6, which is characterized in that the field data mapping block packet
It includes:
First map unit, for mapping data volume size will to be preset in the target dbf file since the header information
Data map to specified memory, wherein the default mapping data volume size is equal to preset data amount size and the head
The sum of data volume size of information；
Field block determination unit, the data in the data for that will map to the specified memory in addition to the header information are true
It is set to present field block.
8. large data files reading device according to claim 6, which is characterized in that the large data files reading device
Further include:
Header information removing module obtains new target dbf file for deleting the header information of the target dbf file；
The field data mapping block is specifically used for: will be located at the preset data amount size on head in the target dbf file
Field data map in specified memory as present field block.
9. a kind of computer equipment, including memory, processor and storage are in the memory and can be in the processor
The computer program of upper operation, which is characterized in that the processor realized when executing the computer program as claim 1 to
Described in any one of 5 the step of large data files read method.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In realization large data files reading side as described in any one of claims 1 to 5 when the computer program is executed by processor
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811203112.1A CN109284260B (en) | 2018-10-16 | 2018-10-16 | Big data file reading method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811203112.1A CN109284260B (en) | 2018-10-16 | 2018-10-16 | Big data file reading method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109284260A true CN109284260A (en) | 2019-01-29 |
| CN109284260B CN109284260B (en) | 2023-10-13 |
Family
ID=65177219
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811203112.1A Active CN109284260B (en) | 2018-10-16 | 2018-10-16 | Big data file reading method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109284260B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111752814A (en) * | 2020-05-26 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Batch processing method and system for RMT (remote metering test) data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7293032B1 (en) * | 2002-12-11 | 2007-11-06 | Ncr Corp. | Compressing decimal types |
| US7543019B1 (en) * | 2004-03-31 | 2009-06-02 | Emc Corporation | Methods and apparatus providing backward compatibility for applications that access a changing object model |
| US20140207771A1 (en) * | 2013-01-21 | 2014-07-24 | Snap-On Incorporated | Methods and Systems for Mapping Repair Orders within a Database |
| CN105446991A (en) * | 2014-07-07 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Data storage method, query method and device |
| CN107797883A (en) * | 2016-09-06 | 2018-03-13 | 南京中兴新软件有限责任公司 | A kind of backup of memory database, restoration methods and device |
| CN108153919A (en) * | 2018-02-28 | 2018-06-12 | 弘成科技发展有限公司 | DBF data export platform and its deriving method |
| CN108280223A (en) * | 2018-02-09 | 2018-07-13 | 弘成科技发展有限公司 | DBF data for College Enrollment import platform and introduction method |
-
2018
- 2018-10-16 CN CN201811203112.1A patent/CN109284260B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7293032B1 (en) * | 2002-12-11 | 2007-11-06 | Ncr Corp. | Compressing decimal types |
| US7543019B1 (en) * | 2004-03-31 | 2009-06-02 | Emc Corporation | Methods and apparatus providing backward compatibility for applications that access a changing object model |
| US20140207771A1 (en) * | 2013-01-21 | 2014-07-24 | Snap-On Incorporated | Methods and Systems for Mapping Repair Orders within a Database |
| CN105446991A (en) * | 2014-07-07 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Data storage method, query method and device |
| CN107797883A (en) * | 2016-09-06 | 2018-03-13 | 南京中兴新软件有限责任公司 | A kind of backup of memory database, restoration methods and device |
| CN108280223A (en) * | 2018-02-09 | 2018-07-13 | 弘成科技发展有限公司 | DBF data for College Enrollment import platform and introduction method |
| CN108153919A (en) * | 2018-02-28 | 2018-06-12 | 弘成科技发展有限公司 | DBF data export platform and its deriving method |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111752814A (en) * | 2020-05-26 | 2020-10-09 | 苏州浪潮智能科技有限公司 | Batch processing method and system for RMT (remote metering test) data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109284260B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110647582B (en) | Method, device, storage medium and computer equipment for block chain network consensus verification | |
| CN109189785A (en) | Date storage method, device, computer equipment and storage medium | |
| CN109542961A (en) | Date storage method, device, computer equipment and storage medium | |
| CN109542428A (en) | Method for processing business, device, computer equipment and storage medium | |
| CN105468293A (en) | Method and system for improving flash storage utilization by predicting bad m-pages | |
| CN109597571A (en) | Date storage method, method for reading data, device and computer equipment | |
| CN108834086A (en) | Method, apparatus, computer equipment and the storage medium that short message is sent | |
| CN109471843A (en) | A kind of metadata cache method, system and relevant apparatus | |
| CN109460438A (en) | Message data storage method, device, computer equipment and storage medium | |
| CN109040191A (en) | Document down loading method, device, computer equipment and storage medium | |
| CN108512948B (en) | Address book updating method and device, computer equipment and storage medium | |
| CN109271461A (en) | The increment synthesized backup method and device of SQL Server database | |
| CN109542346A (en) | Dynamic data cache allocation method, device, computer equipment and storage medium | |
| CN115686932A (en) | Backup set file recovery method and device and computer equipment | |
| CN109325026A (en) | Data processing method, device, equipment and medium based on big data platform | |
| CN109284260A (en) | Large data files read method, device, computer equipment and storage medium | |
| CN111208941A (en) | File management method and device, computer equipment and computer readable storage medium | |
| CN104346298B (en) | Data processing method, device and smart card based on smart card | |
| CN111966531B (en) | Data snapshot method and device, computer equipment and storage medium | |
| CN116225314A (en) | Data writing method, device, computer equipment and storage medium | |
| CN113704027B (en) | File aggregation compatible method and device, computer equipment and storage medium | |
| CN109634875A (en) | Order data method for caching and processing, device, computer equipment and storage medium | |
| CN109542872A (en) | Method for reading data, device, computer equipment and storage medium | |
| CN115543871A (en) | Data storage method and related equipment | |
| CN110008140A (en) | EMS memory management process, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |