CN103593440A - Method and device for reading and writing log file - Google Patents
Method and device for reading and writing log file Download PDFInfo
- Publication number
- CN103593440A CN103593440A CN201310575219.XA CN201310575219A CN103593440A CN 103593440 A CN103593440 A CN 103593440A CN 201310575219 A CN201310575219 A CN 201310575219A CN 103593440 A CN103593440 A CN 103593440A
- Authority
- CN
- China
- Prior art keywords
- daily record
- data
- record data
- read
- temporary table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000001514 detection method Methods 0.000 claims abstract description 11
- 238000001914 filtration Methods 0.000 claims description 10
- 230000007246 mechanism Effects 0.000 abstract description 10
- 230000000694 effects Effects 0.000 abstract description 6
- 238000003780 insertion Methods 0.000 abstract 1
- 230000037431 insertion Effects 0.000 abstract 1
- 230000008569 process Effects 0.000 description 18
- 241001269238 Data Species 0.000 description 13
- 230000008878 coupling Effects 0.000 description 8
- 238000010168 coupling process Methods 0.000 description 8
- 238000005859 coupling reaction Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000004364 calculation method Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008602 contraction Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000004321 preservation Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 238000013497 data interchange Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1858—Parallel file systems, i.e. file systems supporting multiple processors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and device for reading and writing a log file. The method comprises the steps of reading, wherein read log data in the log file are stored a corresponding internal storage set; detection, wherein whether the number of lines of the internal storage set is larger than a preset threshold value or not is detected; insertion, wherein the log data in the internal storage set are inserted into a first temporary table in a database under the condition that the number of the lines of the internal storage set is larger than the preset threshold value; return, wherein the step of reading is executed again until the log data are all stored in the first temporary table; updating, wherein data in the first temporary table are used for updating a formal table of the database. The method and device solve the problems that in the prior art, the time of importing the log file into the database is long and efficiency is low, the effect of carrying out reading and writing simultaneously is achieved, the parallel processing mechanism is used, the overall processing time of a program is shortened, and performance is improved.
Description
Technical field
The present invention relates to data processing field, in particular to a kind of reading/writing method and device of journal file.
Background technology
In prior art, the common way of journal file importing database is, by program, read line by line journal file the write memory line by line of specified format, handle after this journal file, just internal memory one-time write database, also first read file and just understood write into Databasce, adopted the time of the time+write into Databasce of time=file reading that this scheme program carries out.The in the situation that of having mass data in journal file, the time of file reading and the execution time of write into Databasce increase considerably respectively, cause the execution time of program integral body also to increase considerably.Adopt said method because carrying out flow process is serial linearity, in program file reading process, completing of read operation waited in the operation of write into Databasce always, caused a large amount of resources idles.Meanwhile, in the time of enough large of the data of once reading in, can cause the database write angle of incidence long the whole write into Databasces of these data, in the short time, the daily record of database is risen suddenly and sharply, if write failure, the time of rollback mass data is also very long.
, inefficient problem long for the time that in prior art, journal file is imported to database, not yet propose effective solution at present.
Summary of the invention
, inefficient problem long for the time that in correlation technique, journal file is imported to database, not yet propose effective solution at present, and for this reason, fundamental purpose of the present invention is to provide a kind of reading/writing method and device of journal file, to address the above problem.
To achieve these goals, according to an aspect of the present invention, provide a kind of reading/writing method of journal file, the method comprises: read step: the daily record data reading in journal file is preserved into corresponding internal memory set; Detecting step: whether the line number that detects internal memory set is greater than predetermined threshold value; Inserting step: in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set; Return to step: return and carry out read step until daily record data is all preserved in the first temporary table; Step of updating: the formal table that uses the Data Update database in the first temporary table.
Further, read step comprises: from journal file, read line by line the first daily record data; The first daily record data is carried out to filtration treatment and obtain the second daily record data; The second daily record data unserializing is obtained to object elements; Object elements is preserved into internal memory set.
Whether further, after detecting step, method also comprises: in the situation that the line number of internal memory set is not more than predetermined threshold value, detects and the daily record data in journal file has been read; In the situation that the daily record data in journal file has been read, carry out inserting step; In the situation that the daily record data in journal file has not been read, carry out and return to step.
Further, step of updating comprises: the data in the first temporary table are carried out to duplicate removal processing and obtain the second temporary table; Use the Data Update in the second temporary table formally to show.
Further, method also comprises: delete the data in the first temporary table and the second temporary table.
To achieve these goals, according to a further aspect in the invention, provide a kind of read-write equipment of journal file, this device comprises: read module, for the daily record data that reads journal file being preserved into corresponding internal memory set; Whether detection module, be greater than predetermined threshold value for detection of the line number of internal memory set; Insert module, in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set; Return to execution module, for returning, carry out read step until daily record data is all preserved into the first temporary table; Update module, for being used the formal table of the Data Update database of the first temporary table.
Further, read module comprises: reading submodule, for reading line by line the first daily record data from journal file; Filtering module, obtains the second daily record data for the first daily record data is carried out to filtration treatment; Unserializing module, for obtaining object elements by the second daily record data unserializing; Preserve module, for object elements is preserved into internal memory set.
Further, device also comprises: whether detection sub-module, in the situation that the line number of internal memory set is not more than predetermined threshold value, detect and the daily record data in journal file read; The first execution module, in the situation that the daily record data of journal file has been read, carries out insert module; The second execution module, in the situation that the daily record data of journal file has not been read, carries out and returns to execution module.
Further, step of updating comprises: duplicate removal module, obtains the second temporary table for the data of the first temporary table being carried out to duplicate removal processing; Upgrade submodule, for using the Data Update of the second temporary table formally to show.
Further, device also comprises: removing module, and for deleting the data of the first temporary table and the second temporary table.
Adopt the present invention, to read after daily record data in journal file preserves corresponding internal memory set, whether the line number that can detect internal memory set is greater than predetermined threshold value, and in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set, then return to carry out and read the step of daily record data until daily record data is all preserved in the first temporary table, after in all daily record datas are all preserved to the first temporary table, use the Data Update in the first temporary table formally to show.In this log processing logic, realized the mechanism of parallel processing, a large amount of daily record datas be split as a plurality of small amount of data intert main thread read process in the renewal that walks abreast, also can be when main thread reads daily record data, sub-thread is by daily record data write into Databasce, upgrade the data in formal table, realized the effect of read-while-write, use the mechanism of this parallel processing, shortened the program bulk treatment time, improved performance and solved and in prior art, journal file is imported to long, inefficient problem of time of database.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide a further understanding of the present invention, forms the application's a part, and schematic description and description of the present invention is used for explaining the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings:
Fig. 1 is the read-write equipment schematic diagram according to the journal file of the embodiment of the present invention;
Fig. 2 is according to the process flow diagram of the reading/writing method of the journal file of the embodiment of the present invention; And
Fig. 3 is according to the process flow diagram of the reading/writing method of a kind of optional journal file of the embodiment of the present invention.
Embodiment
First, the part noun or the term that in the process that the embodiment of the present invention is described, occur are applicable to description below:
Rollback (Rollback): the failure of rollback general reference program updates, return to the behavior of last correct status.
Serializing (Serialize): the process that Obj State is converted to the form that can keep or transmit.
Unserializing (Deserialize): the process that data stream is changed into object.
Regular expression (Regular Expression): regular expression is to describe, mate a series of character strings that meet certain syntactic rule by single character string, in a lot of text editors, regular expression is usually used to retrieval, replaces the text that those meet certain pattern.
JSON (JavaScript Object Notation): a kind of data interchange format of lightweight.
Major key (Primary Key): be the one or more fields in table, its value is used for a certain record of label table uniquely.
Hash (Hash): be exactly that (be called again pre-mapping, pre-image), by hashing algorithm, be transformed into the output of regular length, this output is exactly hashed value the input of random length.This conversion is a kind of compressing mapping, and namely, the space of hashed value is conventionally much smaller than the space of input, and different inputs may hash to identical output, and can not carry out unique definite input value from hashed value.It is exactly briefly a kind of function that the message compression of random length is arrived to the eap-message digest of a certain regular length.
In order to make those skilled in the art person understand better the present invention program, below in conjunction with the accompanying drawing in the embodiment of the present invention, technical scheme in the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the embodiment of a part of the present invention, rather than whole embodiment.Embodiment based in the present invention, those of ordinary skills, not making the every other embodiment obtaining under creative work prerequisite, should belong to the scope of protection of the invention.
It should be noted that, the term " first " in instructions of the present invention and claims and above-mentioned accompanying drawing, " second " etc. are for distinguishing similar object, and needn't be for describing specific order or precedence.The data that should be appreciated that such use suitably can exchanged in situation, so as embodiments of the invention described herein can with except diagram here or describe those order enforcement.In addition, term " comprises " and " having " and their any distortion, intention is to cover not exclusive comprising, for example, those steps or unit that the process that has comprised series of steps or unit, method, system, product or equipment are not necessarily limited to clearly list, but can comprise clearly do not list or for these processes, method, product or equipment intrinsic other step or unit.
Fig. 1 is the read-write equipment schematic diagram according to the journal file of the embodiment of the present invention.As shown in Figure 1, this device can comprise: read module 10, for the daily record data that reads journal file being preserved into corresponding internal memory set; Whether detection module 30, be greater than predetermined threshold value for detection of the line number of internal memory set; Insert module 50, in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set; Return to execution module 70, for returning, carry out read step until daily record data is all preserved into the first temporary table; Update module 90, for being used the formal table of the Data Update database of the first temporary table.
Adopt the present invention, to read after daily record data in journal file preserves corresponding internal memory set, whether the line number that can detect internal memory set is greater than predetermined threshold value, and in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set, then return to carry out and read the step of daily record data until daily record data is all preserved in the first temporary table, after in all daily record datas are all preserved to the first temporary table, use the Data Update in the first temporary table formally to show.In this log processing logic, realized the mechanism of parallel processing, a large amount of daily record datas be split as a plurality of small amount of data intert main thread read process in the renewal that walks abreast, also can be when main thread reads daily record data, sub-thread is by daily record data write into Databasce, upgrade the data in formal table, realized the effect of read-while-write, use the mechanism of this parallel processing, shortened the program bulk treatment time, improved performance, solved and in prior art, journal file has been imported to the time length of database, inefficient problem.
Further, read module comprises: reading submodule, for reading line by line the first daily record data from journal file; Filtering module, obtains the second daily record data for the first daily record data is carried out to filtration treatment; Unserializing module, for obtaining object elements by the second daily record data unserializing; Preserve module, for object elements is preserved into internal memory set.
Particularly, before carrying out above-described embodiment of the application, data acquisition in can initialization internal memory and carry out the built-in variable that this embodiment need to use and (after completing the initialization of internal memory, open journal file, read line by line the first daily record data in journal file.Daily record data in this embodiment and the first daily record data are same data.
Wherein, this built-in variable refers in program the temporary variable for exchanges data or transmission, only uses program is inner.
More specifically, by regular expression, from the first daily record data reading, filter out and do not meet regular data stream (data stream that the first daily record data forms) and obtain the second daily record data.In this step, the form of expression of daily record data is character string.
By this step, can reduce and not meet in a large number regular daily record data, thereby can effectively reduce the quantity that follow-up data is processed, reduced resource occupation and can greatly improve processing speed.
Particularly, daily record data is put into internal memory set, daily record is read in circulation afterwards, until that daily record is all read is complete.Daily record data takes the form of object elements in this embodiment.
More specifically, in this preservation process, the daily record data with different attribute can be preserved in different internal memory set, also can daily record data be preserved into corresponding internal memory set according to the attribute information of daily record data.Wherein, attribute information can comprise: the information such as type of service, user profile and access time.
Further, device also comprises: whether detection sub-module, in the situation that the line number of internal memory set is not more than predetermined threshold value, detect and the daily record data in journal file read; The first execution module, in the situation that the daily record data of journal file has been read, carries out insert module; The second execution module, in the situation that the daily record data of journal file has not been read, carries out and returns to execution module.
Further, step of updating comprises: duplicate removal module, obtains the second temporary table for the data of the first temporary table being carried out to duplicate removal processing; Upgrade submodule, for using the Data Update of the second temporary table formally to show.
Further, device also comprises: removing module, and for deleting the data of the first temporary table and the second temporary table.
Owing to having a large amount of repeating datas in the first temporary table, the mass data repeating is directly deleted and generated the second temporary table in the first temporary table, then use the data in the formal table of Data Update in the second temporary table, compared with directly deleting a large amount of repeating datas after in the formal table of the data importing in the first temporary table, can save a large amount of time, adopt this scheme can greatly improve treatment effeciency.
In the above embodiment of the present invention, delete the data in the first temporary table and the second temporary table, can contraction data storehouse releasing resource.
Particularly, the hash value that the temporary table after duplicate removal (in the second temporary table) can be generated according to the each row of data primary key column in the second temporary table, data to temporary table and formal table are carried out Hash coupling line by line, the data of temporary table Already in formally table the Data Update of temporary table in formal table, the data of temporary table are not present in formal table and the data of temporary table are inserted in formal table.
In the above-described embodiments, the data in the second temporary table and formal table being carried out to Hash coupling line by line can realize particularly by the following method:
Relatively in two tables, whether the hash value of each row of data is identical, in the situation that hash value is identical, confirms that the match is successful; In the situation that hash is different, confirm that coupling is unsuccessful.
In the situation that the match is successful, confirm that the data of this row in the second temporary table exist in formal table, use the value of the data of corresponding row in the formal table of data modification in the second temporary table; In the unsuccessful situation of coupling, confirm that the data of this row in the second temporary table do not exist in formal table, the data of changing one's profession in the second temporary table are inserted in formal table.
Detection sub-module in above-described embodiment, the first execution module, the second execution module, duplicate removal module, renewal submodule and removing module can be used the method in following embodiment of the method to realize.
Fig. 2 is that the method comprises the steps: as shown in Figure 2 according to the process flow diagram of the reading/writing method of the journal file of the embodiment of the present invention
Step S102, read step: the daily record data reading in journal file is preserved into corresponding internal memory set.
Step S104, detecting step: whether the line number that detects internal memory set is greater than predetermined threshold value.
Step S106, inserting step: in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set.
Step S108, returns to step: return and carry out read step until daily record data is all preserved in the first temporary table.
Step S110, step of updating: the formal table that uses the Data Update database in the first temporary table.
Adopt the present invention, to read after daily record data in journal file preserves corresponding internal memory set, whether the line number that can detect internal memory set is greater than predetermined threshold value, and in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set, then return to carry out and read the step of daily record data until daily record data is all preserved in the first temporary table, after in all daily record datas are all preserved to the first temporary table, use the Data Update in the first temporary table formally to show.In this log processing logic, realized the mechanism of parallel processing, a large amount of daily record datas be split as a plurality of small amount of data wear tear open main thread read process in the renewal that walks abreast, also can be when main thread reads daily record data, sub-thread is by daily record data write into Databasce, upgrade the data in formal table, realized the effect of read-while-write, use the mechanism of this parallel processing, shortened the program bulk treatment time, improved performance and solved and in prior art, journal file is imported to long, inefficient problem of time of database.
In the above embodiment of the present invention, by the mechanism of read-while-write parallel processing, reduce resources idle, shorten the whole execution time, improve execution efficiency, by setting the threshold value of a read-write balance, reduce the data volume of one-time write database, improve the readwrite performance of system.
According to the abovementioned embodiments of the present invention, read step can comprise: from journal file, read line by line the first daily record data; The first daily record data is carried out to filtration treatment and obtain the second daily record data; The second daily record data unserializing is obtained to object elements; Object elements is preserved into internal memory set.
Particularly, as shown in Figure 3, the above embodiment of the present invention can comprise the steps:
Step S201: read daily record data.
Step S202: read next daily record data.
Particularly, before carrying out above-described embodiment of the application, data acquisition in can initialization internal memory and carry out the built-in variable that this embodiment need to use and (after completing the initialization of internal memory, open journal file, read line by line the first daily record data in journal file.Daily record data in this embodiment and the first daily record data are same data.
Wherein, this built-in variable refers in program the temporary variable for exchanges data or transmission, only uses program is inner.
Step S203: detect next daily record data and whether meet regular expression.By this step, realize in above-described embodiment the first daily record data is carried out to the step that filtration treatment obtains the second daily record data.
Wherein, particularly, in the situation that next daily record data meets regular expression, show that this daily record data meets rule, by the processing of this daily record data (daily record data is herein the second daily record data) execution step S205.In the situation that next daily record data meets regular expression, execution step S204: throw exception, and return to execution step S202.
More specifically, by regular expression, from the first daily record data reading, filter out and do not meet regular data stream (data stream that the first daily record data forms) and obtain the second daily record data.In this step, the form of expression of daily record data is character string.
By this step, can reduce and not meet in a large number regular daily record data, thereby can effectively reduce the quantity that follow-up data is processed, reduced resource occupation and can greatly improve processing speed.
Step S205: daily record data unserializing is become to object elements.
Particularly, according to preset rules, the second daily record data (the first daily record data can be character string in the present embodiment) is deserialized as to object elements.
Wherein, the rule in above-described embodiment can be that the data layout of client and server commitment is, parameters such as the title of data item and type definitions; Data stream can be scale, for example: 010101, JSON:{ " name ": " john ", " age ": 24}, parametric form: sn=85 & gsve=1.3.1.0 etc., the form of transmission daily record data that client and server end is made an appointment and the title of data item and type definition are the preset rules in above-described embodiment.
Particularly, data receiver (being the server in above-described embodiment) has defined data item number, title and the type that receives data, for example: A: numeric type, B: character type, C: character type, client sends and must send data by this definition, if data item number, title and type any one do not conform to symbol definition, is all considered to not meet rule.
Step S206: daily record data is preserved into internal memory set.
Particularly, daily record data is put into internal memory set, daily record is read in circulation afterwards, until that daily record is all read is complete.Daily record data takes the form of object elements in this embodiment.
More specifically, in this preservation process, the daily record data with different attribute can be preserved in different internal memory set, also can daily record data be preserved into corresponding internal memory set according to the attribute information of daily record data.Wherein, attribute information can comprise: the information such as type of service, user profile and access time.
Step S207: whether the line number that detects internal memory set is greater than predetermined threshold value.
Wherein, in the situation that line number is greater than predetermined threshold value, execution step S209; In the situation that line number is not more than predetermined threshold value, execution step S208.
Step S208: detect daily record data and whether all run through.Wherein, in the situation that daily record data all runs through, execution step S209; In the situation that daily record data does not all run through, return to execution step S202, until all daily record datas are run through.
Step S209: daily record data is inserted to temporary table.
Particularly, internal memory set of rows number is arranged to a predetermined threshold value, in the line number of internal memory set, be greater than when threshold value is set, open a thread, the indications of mission thread is from adding one, by database script language first temporary table (middle table) in the data in internal memory set (i.e. the second daily record data) data inserting storehouse.
Wherein, the list structure of temporary table and formal table are in full accord.
Step S210: detect daily record data and whether successfully insert temporary table.Temporary table in this step is the first temporary table in above-described embodiment.
Particularly, if insert unsuccessfully rollback affairs, throw exception termination routine; If insert successfully, perform step S211, meanwhile, and the resource of sub-thread release busy, mission thread indications is from subtracting one.
Step S211: detect daily record data and whether run through.Wherein, in the situation that daily record data has not read, return to execution step S202; In the situation that daily record data has read, execution step S212.
Particularly, main thread continues to read daily record data, until that daily record data all reads is complete, the quantity by mission thread indications judges whether to also have sub-thread not complete, and when mission thread indications equals zero, sub-thread is all finished.
Whether, in the above embodiment of the present invention, after detecting step, method can also comprise: in the situation that the line number of internal memory set is not more than predetermined threshold value, detects and the daily record data in journal file has been read; In the situation that the daily record data in journal file has been read, carry out inserting step; In the situation that the daily record data in journal file has not been read, carry out and return to step.
According to the abovementioned embodiments of the present invention, importing step can comprise: the data in the first temporary table are carried out to duplicate removal processing and obtain the second temporary table; Data importing in the second temporary table is formally shown.Particularly, the second temporary table can be and the first temporary table
Owing to having a large amount of repeating datas in the first temporary table, the mass data repeating is directly deleted and generated the second temporary table in the first temporary table, then use the data in the formal table of Data Update in the second temporary table, compared with directly deleting a large amount of repeating datas after in the formal table of the data importing in the first temporary table, can save a large amount of time, adopt this scheme can greatly improve treatment effeciency.
Particularly, after importing step, method can also comprise: delete the data in the first temporary table and the second temporary table.
Delete the data in the first temporary table and the second temporary table, can contraction data storehouse releasing resource.
Particularly, step S212 shown in Figure 3 can realize above-described embodiment to step S213:
Step S212: the data importing database in temporary table is formally shown.
Particularly, main thread waits for that sub-thread is all finished (importing that now each sub-thread has completed temporary table data separately), the data importing in all temporary tables in the formal table of database.
More specifically, the hash value that the temporary table after duplicate removal (in the second temporary table) can be generated according to the each row of data primary key column in the second temporary table, data to temporary table and formal table are carried out Hash coupling line by line, the data of temporary table Already in formally table the Data Update of temporary table in formal table, the data of temporary table are not present in formal table and the data of temporary table are inserted in formal table.
In the above-described embodiments, the data in the second temporary table and formal table being carried out to Hash coupling line by line can realize particularly by the following method:
Relatively in two tables, whether the hash value of each row of data is identical, in the situation that hash value is identical, confirms that the match is successful; In the situation that hash is different, confirm that coupling is unsuccessful.
In the situation that the match is successful, confirm that the data of this row in the second temporary table exist in formal table, use the value of the data of corresponding row in the formal table of data modification in the second temporary table; In the unsuccessful situation of coupling, confirm that the data of this row in the second temporary table do not exist in formal table, the data of changing one's profession in the second temporary table are inserted in formal table.
Step S213: whether the formal table of data importing detecting in temporary table is failed.
In the situation that the data importing in temporary table is formally shown is failed, rollback affairs, throw exception misregistration information termination routine; If import successfully, submit affairs to.
The formal table of mentioning is in embodiments of the present invention the table that is kept in advance predetermined format in database.
It should be noted that, in the step shown in the process flow diagram of accompanying drawing, can in the computer system such as one group of computer executable instructions, carry out, and, although there is shown logical order in flow process, but in some cases, can carry out shown or described step with the order being different from herein.
Formal table in the above embodiment of the present invention can be the table for save data of the set form that pre-stores in database.
From above description, can find out, the present invention has realized following technique effect: adopt the present invention, to read after daily record data in journal file preserves corresponding internal memory set, whether the line number that can detect internal memory set is greater than predetermined threshold value, and in the situation that the line number of internal memory set is greater than predetermined threshold value, by the first temporary table in the daily record data data inserting storehouse in internal memory set, then return to carry out and read the step of daily record data until daily record data is all preserved in the first temporary table, after in all daily record datas are all preserved to the first temporary table, use the Data Update in the first temporary table formally to show.In this log processing logic, realized the mechanism of parallel processing, a large amount of daily record datas be split as a plurality of small amount of data wear tear open main thread read process in the renewal that walks abreast, also can be when main thread reads daily record data, sub-thread is by daily record data write into Databasce, upgrade the data in formal table, realized the effect of read-while-write, use the mechanism of this parallel processing, shortened the program bulk treatment time, improved performance and solved and in prior art, journal file is imported to long, inefficient problem of time of database.
Obviously, those skilled in the art should be understood that, above-mentioned each module of the present invention or each step can realize with general calculation element, they can concentrate on single calculation element, or be distributed on the network that a plurality of calculation elements form, alternatively, they can be realized with the executable program code of calculation element, thereby, they can be stored in memory storage and be carried out by calculation element, or they are made into respectively to each integrated circuit modules, or a plurality of modules in them or step are made into single integrated circuit module to be realized.Like this, the present invention is not restricted to any specific hardware and software combination.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.
Claims (10)
1. a reading/writing method for journal file, is characterized in that, comprising:
Read step: the daily record data reading in journal file is preserved into corresponding internal memory set;
Detecting step: whether the line number that detects described internal memory set is greater than predetermined threshold value;
Inserting step: in the situation that the described line number of described internal memory set is greater than described predetermined threshold value, by the first temporary table in the described daily record data data inserting storehouse in described internal memory set;
Return to step: return and carry out described read step until described daily record data is all preserved in described the first temporary table;
Step of updating: the formal table that uses database described in the Data Update in described the first temporary table.
2. reading/writing method according to claim 1, is characterized in that, described read step comprises:
From described journal file, read line by line the first daily record data;
Described the first daily record data is carried out to filtration treatment and obtain the second daily record data;
Described the second daily record data unserializing is obtained to object elements;
Described object elements is preserved into described internal memory set.
3. reading/writing method according to claim 1, is characterized in that, after described detecting step, described method also comprises:
In the situation that the described line number of described internal memory set is not more than described predetermined threshold value, detects and whether the described daily record data in described journal file has been read;
In the situation that the described daily record data in described journal file has been read, carry out described inserting step;
In the situation that the described daily record data in described journal file has not been read, return to step described in execution.
4. according to the reading/writing method described in any one in claims 1 to 3, it is characterized in that, described step of updating comprises:
Data in described the first temporary table are carried out to duplicate removal processing and obtain the second temporary table;
Use described in the Data Update in described the second temporary table formally table.
5. reading/writing method according to claim 4, is characterized in that, after described step of updating, described method also comprises:
Delete the data in described the first temporary table and described the second temporary table.
6. a read-write equipment for journal file, is characterized in that, comprising:
Read module, for preserving the daily record data that reads journal file into corresponding internal memory set;
Whether detection module, be greater than predetermined threshold value for detection of the line number of described internal memory set;
Insert module, in the situation that the described line number of described internal memory set is greater than described predetermined threshold value, by the first temporary table in the described daily record data data inserting storehouse in described internal memory set;
Return to execution module, for returning, carry out described read step until described daily record data is all preserved into described the first temporary table;
Update module, for being used the formal table of database described in the Data Update of described the first temporary table.
7. read-write equipment according to claim 6, is characterized in that, described read module comprises:
Reading submodule, for reading line by line the first daily record data from described journal file;
Filtering module, obtains the second daily record data for described the first daily record data is carried out to filtration treatment;
Unserializing module, for obtaining object elements by described the second daily record data unserializing;
Preserve module, for described object elements is preserved into described internal memory set.
8. read-write equipment according to claim 6, is characterized in that, described device also comprises:
Whether detection sub-module, in the situation that the described line number of described internal memory set is not more than described predetermined threshold value, detect and the described daily record data in described journal file read;
The first execution module, in the situation that the described daily record data of described journal file has been read, carries out described insert module;
The second execution module, in the situation that the described daily record data of described journal file has not been read, returns to execution module described in execution.
9. according to the read-write equipment described in any one in claim 6 to 8, it is characterized in that, described step of updating comprises:
Duplicate removal module, obtains the second temporary table for the data of described the first temporary table being carried out to duplicate removal processing;
Upgrade submodule, for using described in the Data Update of described the second temporary table formally table.
10. read-write equipment according to claim 9, is characterized in that, described device also comprises:
Removing module, for deleting the data of described the first temporary table and described the second temporary table.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310575219.XA CN103593440B (en) | 2013-11-15 | 2013-11-15 | The reading/writing method and device of journal file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310575219.XA CN103593440B (en) | 2013-11-15 | 2013-11-15 | The reading/writing method and device of journal file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103593440A true CN103593440A (en) | 2014-02-19 |
| CN103593440B CN103593440B (en) | 2017-10-27 |
Family
ID=50083581
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310575219.XA Active CN103593440B (en) | 2013-11-15 | 2013-11-15 | The reading/writing method and device of journal file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103593440B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104462396A (en) * | 2014-12-10 | 2015-03-25 | 北京国双科技有限公司 | Method and device for handing character strings |
| CN104809152A (en) * | 2015-03-12 | 2015-07-29 | 杭州米加科技有限公司 | Method and system for saving PostgresSQL shared memory |
| CN105260485A (en) * | 2015-11-20 | 2016-01-20 | 杭州数梦工场科技有限公司 | Method and device for loading data |
| CN105430534A (en) * | 2015-10-29 | 2016-03-23 | Tcl集团股份有限公司 | Intelligent device reporting data method and system |
| CN105701122A (en) * | 2014-11-28 | 2016-06-22 | 深圳市腾讯计算机系统有限公司 | Log collection method, apparatus and system |
| CN106599222A (en) * | 2016-12-19 | 2017-04-26 | 广州四三九九信息科技有限公司 | Method and equipment for log streaming type parallel processing |
| CN106844526A (en) * | 2016-12-29 | 2017-06-13 | 金蝶软件（中国）有限公司 | Bill processing method and device |
| CN108415892A (en) * | 2018-03-07 | 2018-08-17 | 深圳市易迈数据技术有限公司 | A kind of rapid data analysis generation report processing method |
| CN108446370A (en) * | 2018-03-15 | 2018-08-24 | 苏州思必驰信息科技有限公司 | Voice data statistical method and system |
| CN109213736A (en) * | 2017-06-29 | 2019-01-15 | 阿里巴巴集团控股有限公司 | The compression method and device of log |
| CN109828719A (en) * | 2018-12-15 | 2019-05-31 | 平安科技（深圳）有限公司 | Magnetic disc control method, device and relevant device where commitLog file based on cloud monitoring |
| CN111488237A (en) * | 2020-05-15 | 2020-08-04 | 征图新视(江苏)科技股份有限公司 | Quick verification method for two-dimensional code big data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1932812A (en) * | 2005-09-16 | 2007-03-21 | 腾讯科技（深圳）有限公司 | Method and apparatus for holding journal |
| US20110283139A1 (en) * | 2005-12-29 | 2011-11-17 | Trika Sanjeev N | Method and apparatus to maintain data integrity in disk cache memory during and after periods of cache inaccessibility |
| CN102542054A (en) * | 2011-12-29 | 2012-07-04 | 厦门雅迅网络股份有限公司 | Method utilizing cache tables to improve insertion performance of data in database |
| CN102724279A (en) * | 2012-05-15 | 2012-10-10 | 北京首钢自动化信息技术有限公司 | System for realizing log-saving and log-managing |
| CN103049533A (en) * | 2012-12-23 | 2013-04-17 | 北京人大金仓信息技术股份有限公司 | Method for quickly loading data into database |
| CN103064908A (en) * | 2012-12-18 | 2013-04-24 | 北京讯鸟软件有限公司 | Method for rapidly removing repeated list through a memory |
-
2013
- 2013-11-15 CN CN201310575219.XA patent/CN103593440B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1932812A (en) * | 2005-09-16 | 2007-03-21 | 腾讯科技（深圳）有限公司 | Method and apparatus for holding journal |
| US20110283139A1 (en) * | 2005-12-29 | 2011-11-17 | Trika Sanjeev N | Method and apparatus to maintain data integrity in disk cache memory during and after periods of cache inaccessibility |
| CN102542054A (en) * | 2011-12-29 | 2012-07-04 | 厦门雅迅网络股份有限公司 | Method utilizing cache tables to improve insertion performance of data in database |
| CN102724279A (en) * | 2012-05-15 | 2012-10-10 | 北京首钢自动化信息技术有限公司 | System for realizing log-saving and log-managing |
| CN103064908A (en) * | 2012-12-18 | 2013-04-24 | 北京讯鸟软件有限公司 | Method for rapidly removing repeated list through a memory |
| CN103049533A (en) * | 2012-12-23 | 2013-04-17 | 北京人大金仓信息技术股份有限公司 | Method for quickly loading data into database |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105701122A (en) * | 2014-11-28 | 2016-06-22 | 深圳市腾讯计算机系统有限公司 | Log collection method, apparatus and system |
| CN105701122B (en) * | 2014-11-28 | 2020-06-16 | 深圳市腾讯计算机系统有限公司 | Log collection method, device and system |
| CN104462396B (en) * | 2014-12-10 | 2017-12-19 | 北京国双科技有限公司 | Character string processing method and device |
| CN104462396A (en) * | 2014-12-10 | 2015-03-25 | 北京国双科技有限公司 | Method and device for handing character strings |
| CN104809152A (en) * | 2015-03-12 | 2015-07-29 | 杭州米加科技有限公司 | Method and system for saving PostgresSQL shared memory |
| CN104809152B (en) * | 2015-03-12 | 2018-08-17 | 杭州米加科技有限公司 | A kind of method and system for saving PostgreSQL shared drives |
| CN105430534A (en) * | 2015-10-29 | 2016-03-23 | Tcl集团股份有限公司 | Intelligent device reporting data method and system |
| CN105430534B (en) * | 2015-10-29 | 2019-12-10 | Tcl集团股份有限公司 | method and system for reporting data by intelligent equipment |
| CN105260485B (en) * | 2015-11-20 | 2019-05-31 | 杭州数梦工场科技有限公司 | A kind of method and apparatus of data load |
| CN105260485A (en) * | 2015-11-20 | 2016-01-20 | 杭州数梦工场科技有限公司 | Method and device for loading data |
| CN106599222A (en) * | 2016-12-19 | 2017-04-26 | 广州四三九九信息科技有限公司 | Method and equipment for log streaming type parallel processing |
| CN106844526B (en) * | 2016-12-29 | 2019-12-10 | 金蝶软件（中国）有限公司 | document processing method and device |
| CN106844526A (en) * | 2016-12-29 | 2017-06-13 | 金蝶软件（中国）有限公司 | Bill processing method and device |
| CN109213736A (en) * | 2017-06-29 | 2019-01-15 | 阿里巴巴集团控股有限公司 | The compression method and device of log |
| CN108415892A (en) * | 2018-03-07 | 2018-08-17 | 深圳市易迈数据技术有限公司 | A kind of rapid data analysis generation report processing method |
| CN108446370A (en) * | 2018-03-15 | 2018-08-24 | 苏州思必驰信息科技有限公司 | Voice data statistical method and system |
| CN109828719A (en) * | 2018-12-15 | 2019-05-31 | 平安科技（深圳）有限公司 | Magnetic disc control method, device and relevant device where commitLog file based on cloud monitoring |
| CN109828719B (en) * | 2018-12-15 | 2022-04-01 | 平安科技（深圳）有限公司 | CommitLog file located disk control method and device based on cloud monitoring and related equipment |
| CN111488237A (en) * | 2020-05-15 | 2020-08-04 | 征图新视(江苏)科技股份有限公司 | Quick verification method for two-dimensional code big data |
| CN111488237B (en) * | 2020-05-15 | 2023-09-22 | 征图新视(江苏)科技股份有限公司 | Quick verification method for big data of two-dimension code |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103593440B (en) | 2017-10-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103593440A (en) | Method and device for reading and writing log file | |
| CN108628942B (en) | Block chain node equipment and data reading and writing method of distributed database | |
| CN111046034B (en) | Method and system for managing memory data and maintaining data in memory | |
| CN110399388A (en) | Data query method, system and equipment | |
| CN109885614B (en) | Data synchronization method and device | |
| US9299082B2 (en) | Fast streams and substreams | |
| CN103593442A (en) | Duplication eliminating method and device for log data | |
| EP3080723A1 (en) | Building features and indexing for knowledge-based matching | |
| CN105740337A (en) | Rapid event matching method in content-based publishing subscription system | |
| CN111444192B (en) | Method, device and equipment for generating Hash of global state in block chain type account book | |
| JP2017526021A (en) | Error correction apparatus and method for data retrieval | |
| CN111444196B (en) | Method, device and equipment for generating Hash of global state in block chain type account book | |
| CN103885721A (en) | Data storing or reading method and device for key-value system | |
| CN110457348B (en) | Data processing method and device | |
| US8407255B1 (en) | Method and apparatus for exploiting master-detail data relationships to enhance searching operations | |
| CN111290714B (en) | Data reading method and device | |
| CN105389394A (en) | Data request processing method and device based on a plurality of database clusters | |
| CN117093619A (en) | Rule engine processing method and device, electronic equipment and storage medium | |
| CN112883088B (en) | Data processing method, device, equipment and storage medium | |
| CN109101595B (en) | Information query method, device, equipment and computer readable storage medium | |
| CN109408035B (en) | Flow configuration method, storage medium and server of business system | |
| CN111124883A (en) | Test case library introduction method, system and equipment based on tree form | |
| CN111262589A (en) | DFA space compression method and device | |
| CN111125332A (en) | Method, device, equipment and storage medium for calculating TF-IDF value of word | |
| CN113779068B (en) | Data query method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Method and device for reading and writing log file Effective date of registration: 20190531 Granted publication date: 20171027 Pledgee: Shenzhen Black Horse World Investment Consulting Co.,Ltd. Pledgor: BEIJING GRIDSUM TECHNOLOGY Co.,Ltd. Registration number: 2019990000503 |
|
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 100083 No. 401, 4th Floor, Haitai Building, 229 North Fourth Ring Road, Haidian District, Beijing Patentee after: BEIJING GRIDSUM TECHNOLOGY Co.,Ltd. Address before: 100086 Beijing city Haidian District Shuangyushu Area No. 76 Zhichun Road cuigongfandian 8 layer A Patentee before: BEIJING GRIDSUM TECHNOLOGY Co.,Ltd. |
|
| PP01 | Preservation of patent right | ||
| PP01 | Preservation of patent right |
Effective date of registration: 20240604 Granted publication date: 20171027 |