CN104850556B - A kind of method and device of data processing - Google Patents
A kind of method and device of data processing Download PDFInfo
- Publication number
- CN104850556B CN104850556B CN201410053223.4A CN201410053223A CN104850556B CN 104850556 B CN104850556 B CN 104850556B CN 201410053223 A CN201410053223 A CN 201410053223A CN 104850556 B CN104850556 B CN 104850556B
- Authority
- CN
- China
- Prior art keywords
- data
- result data
- event information
- database
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
This application discloses a kind of method and device of data processing, to solve the problems, such as that the treatment effeciency of event information in the prior art is relatively low.After this method real time computation system obtains result data to current event information processing, the Data Identification carried according to the result data, whether the time span for judging the Data Identification corresponding write-in moment to current time is more than setting time length, if, then the result data is written in database, and the Data Identification corresponding write-in moment is updated to current time, the result data is otherwise preserved, and handle next event information.The above method when the time span that moment to current time is written is more than setting time length due to only just writing result data to database, remaining time can handle subsequent event information, therefore the treatment effeciency of event information can under the premise of ensureing that real time computation system is consistent with the data in database, be effectively improved.
Description
Technical field
This application involves field of computer technology more particularly to a kind of method and devices of data processing.
Background technology
In today that computer technology continues to develop, need to carry out the event information of generation in practical application scene real-time
Processing, result data is obtained, and by result data storage in the database, so as to subsequent query.
Specifically, can be received by real time computation system and handle event information in real time, then obtained number of results will be handled
According to write-in database.
For example, in logistics information handles scene, the Shipping Information of certain logistics facility pulls collection of letters breath, sends information with charge free
Deng being all an event information.And result data can be then in some period to certain logistics facility（Such as, the same day）Interior hair
Goods amount pulls the result that receipts amount, the amount of sending with charge free etc. are counted.
Specifically, after logistic information systems generate an event information, then the event information is sent to real-time meter
Calculation system.The event information can then be distributed to some treatment progress of itself and handled by real time computation system, this handle into
Journey is determined according to the attribute information and the correspondence of preset attribute information and Data Identification carried in the event information
The Data Identification of the corresponding result data of the attribute information, further according to the result data of the Data Identification obtained before, to this
The result data of Data Identification is updated, and finally updated result data is written in database.
However, for database, number that data can be written in from the unit interval to the database be it is conditional,
And for real time computation system, each treatment progress in real time computation system is each event information of serial process,
Only after the current corresponding result data of event information is written in database, just can to next event information into
Row processing, therefore, once real time computation system is more than the limit of database in the number of unit interval inbound data library write-in data
System, can result in the accumulation of event information, reduces the treatment effeciency of event information, result even in real time computation system event
Barrier.
For example, for a database, the number maximum per second that data can be written to the database is 10000
It is secondary, it is assumed that each event information can lead to the update of 4 result datas, then real time computation system can only at most be supported per second
The processing of 10000/4=2500 event information.If real time computation system had received 2501 event informations in 1 second,
Due to the limitation of database write-in data（10000 times per second）Caused by real time computation system is per second can only handle 2500 things
Part information will make the accumulation of 2501-2500=1 event information.Obviously, if what real time computation system received in 1 second
Event information is much larger than 2500, will result in a large amount of event information accumulation, reduces the treatment effeciency of event information, even
It can lead to real time computation system failure.
Invention content
The embodiment of the present application provides a kind of method and device of data processing, during solving in the prior art due to unit
Between inbound data library write-in data number exist limitation and event information is caused to be accumulated, the treatment effeciency of event information is relatively low,
The problem of even resulting in real time computation system failure.
A kind of method of data processing provided by the embodiments of the present application, including：
Current event information is handled, obtains result data, and be stored in local；
According to the Data Identification that the result data carries, the Data Identification corresponding write-in moment of record is determined,
At the time of the result data for carrying the Data Identification is written in database by the said write moment for the last time；
Whether the time span for judging said write moment to current time is more than setting time length；
If so, the result data locally preserved is written in database, and the Data Identification is corresponding
The write-in moment is updated to current time；
Otherwise, continue locally preserving the result data, and handle next event information.
A kind of device of data processing provided by the embodiments of the present application, including：
Event processing module for handling current event information, obtains result data, and be stored in local；
Determining module for the Data Identification carried according to the result data, determines the Data Identification pair of record
The result data for carrying the Data Identification is written in database by the write-in moment answered, said write moment for the last time
Moment；
Judgment module, whether the time span for judging said write moment to current time is more than that setting time is long
Degree；
Writing module, for the result data that when the judging result of the judgment module is to be, will locally preserve
It is written in database, and the Data Identification corresponding write-in moment is updated to current time；
The event processing module is additionally operable to, and when the judging result of the judgment module is no, is continued in local preservation
The result data, and next event information is handled.
The embodiment of the present application provides a kind of method and device of data processing, and this method real time computation system is to current event
After information processing obtains result data, according to the Data Identification that the result data carries, judge that the Data Identification of record corresponds to
Write-in moment to current time time span whether be more than setting time length, if so, the result data is written to
In database, and the Data Identification corresponding write-in moment is updated to current time, otherwise continues locally preserving the result
Data, and handle next event information.The above method is since real time computation system is only at the write-in moment to current time
Time span just writes result data to database when being more than setting time length, remaining time real time computation system can be handled
Subsequent event information, therefore can be effectively improved under the premise of ensureing that real time computation system is consistent with the data in database
The treatment effeciency of event information will not cause the accumulation of event information, can effectively reduce and break down due to event information is accumulated
Probability.
Description of the drawings
Attached drawing described herein is used for providing further understanding of the present application, forms the part of the application, this Shen
Illustrative embodiments and their description please do not form the improper restriction to the application for explaining the application.In the accompanying drawings：
Fig. 1 is the process of data processing provided by the embodiments of the present application；
Fig. 2 is the schematic diagram of data handling procedure under normal circumstances provided by the embodiments of the present application；
Fig. 3 is the schematic diagram of data handling procedure under abnormal conditions provided by the embodiments of the present application；
Fig. 4 is the apparatus structure schematic diagram of data processing provided by the embodiments of the present application.
Specific embodiment
Since in practical application scene, for certain business, real time computation system is to current event information
After reason obtains result data, delay a period of time writes result data to again can't cause business very big shadow in database
It rings, therefore, it is entirely acceptable that delay a period of time, which is write result data to again in database, and in this section of delay
In time, real time computation system can handle next event information, can thus break through due to the database unit time
The speed bottle-neck of real time computation system processing event information caused by the number limitation of write-in data, so as to ensure in real time
Under the premise of computing system is consistent with the data in database, event information treatment effeciency is improved, is reduced due to event information heap
Long-pending and failure probability.
Purpose, technical scheme and advantage to make the application are clearer, below in conjunction with the application specific embodiment and
Technical scheme is clearly and completely described in corresponding attached drawing.Obviously, described embodiment is only the application one
Section Example, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not doing
Go out all other embodiments obtained under the premise of creative work, shall fall in the protection scope of this application.
Fig. 1 is the process of data processing provided by the embodiments of the present application, specifically includes following steps：
S101：Current event information is handled, obtains result data, and be stored in local.
In the embodiment of the present application, real time computation system can by the treatment progress of itself to current event information at
Reason, obtains corresponding result data, and be stored in local storage medium, is such as stored in local memory or hard disk.Its
In, attribute information is carried in current event information, Data Identification is carried in result data, can also carry end value, Data Identification
For the unique mark result data.
Specifically, since real time computation system can handle different event informations respectively by multiple treatment progress, to
Concurrent processing event information improves the treatment effeciency of event information, therefore, predeterminable attribute information, place in the embodiment of the present application
Correspondence between reason process and Data Identification this three.Wherein, a treatment progress can correspond to multiple attribute informations, and one
A treatment progress can correspond to multiple Data Identifications, but an attribute information can only correspond to a treatment progress, a Data Identification
A treatment progress can only be corresponded to so that the event information of a treatment progress processing is carries category corresponding with the treatment progress
Property information event information, obtained result data is the result data for carrying the corresponding Data Identification of the treatment progress.That is,
One treatment progress only handles the event information for carrying attribute information corresponding with the treatment progress, and a treatment progress can only obtain
To the result data for carrying Data Identification corresponding with the treatment progress.
So as to which the method that real time computation system handles current event information and obtains result data is specifically as follows：In real time
It is corresponding to be distributed to the attribute information according to the attribute information carried in current event information by computing system for current event information
Treatment progress, the treatment progress is then according to current event information, data mark corresponding to the carrying attribute information locally preserved
The result data of knowledge is updated, and using updated result data as the result handled current event information
Data.
For example, current event information is logistics facility A performs delivery operation in the time T orders for being 1 to order note identification
Information, then the attribute information carried in current event information can be logistics facility identification information（That is, logistics facility A）、
Order note identification, action type（That is, delivery operation）One or more of combination.
Assuming that the attribute information carried in above-mentioned current event information is " logistics facility A+ deliveries operation ", preset attribute
The corresponding treatment progress of information " logistics facility A+ deliveries operation " is 1, and corresponding Data Identification is " logistics facility A delivering amounts ",
Then above-mentioned current event information is distributed to treatment progress 1 by real time computation system, and treatment progress 1 then determines the current event information
The corresponding Data Identification of attribute information " logistics facility A+ deliveries operation " of middle carrying is " logistics facility A delivering amounts ", therefore, is carried
Take the result data that the carrying Data Identification locally preserved is " logistics facility A delivering amounts ".
Assuming that the end value carried in the result data of extraction is n（The delivering amount of logistics facility A for representing current statistic is
n）, then treatment progress 1 end value carried in the result data of extraction is updated to by n+1 according to above-mentioned current event information, and
Using updated result data as the result data handled above-mentioned current event information.
Since real time computation system is also possible to be the service that is made of several calculation servers in practical application scene
Therefore device group system, can add Distributor, and the preset attribute information in Distributor in real time computation system
With the correspondence of calculation server, preset attribute information and treatment progress and Data Identification is corresponding in each calculation server
Relationship.Event information is received by Distributor, and according to the correspondence of preset attribute information and calculation server and
The event information received is sent in real time computation system accordingly by the attribute information carried in the event information received
Calculation server.It is right again by calculation server according to preset attribute information and the correspondence for the treatment of progress and Data Identification
The event information is handled.
S102：According to the Data Identification that the result data carries, the Data Identification corresponding write-in moment of record is determined.
In the embodiment of the present application, real time computation system can be directed to each result data preserved, record the result data
The Data Identification corresponding write-in moment of middle carrying, wherein, the Data Identification corresponding write-in moment should for last will carry
At the time of the result data of Data Identification is written in database.So as to which real time computation system passes through above-mentioned steps S101 processing
After current event information obtains result data and is stored in local, then it can determine that the Data Identification carried in the result data corresponds to
The write-in moment.
S103：Whether the time span for judging write-in moment to current time is more than setting time length, if so, performing
Otherwise step S104 performs step S105.
Wherein, which can be set as needed, such as may be set to 10 seconds.It specifically, can be pre-
It first determines to write result data to the longest delay time of database, and for above-mentioned setting time length to be set as not for business
More than the longest delay time.
S104：The result data locally preserved is written in database, and during write-in that the Data Identification is corresponding
Quarter is updated to current time.
If it is determined that the time span at write-in moment to the current time has been more than setting time length, real time computation system
Then the local result data that is stored in that step S101 is obtained is written in database, and will be taken in the obtained result data
The Data Identification corresponding write-in moment of band is updated to current time.
Wherein, when writing result data in database, knot that real time computation system can be obtained according to step S101
The Data Identification carried in fruit data, the data for the carrying Data Identification for determining to preserve in database, and by determining data
The data preserved in library are directly updated to the result data, the result that can will be also carried in the data preserved in determining database
Value is updated to the end value carried in the result data.
Further, real time computation system can also according to the Data Identification carried in the result data that step S101 is obtained,
Determine to preserve in database carries the data of the Data Identification, and judges that is preserved in database carries the number of the Data Identification
Whether according to identical with the result data that step S101 is obtained, if identical, the result data without step S101 is obtained is written
Database, if it is different, being then the result data by the data update preserved in determining database.
S105：Continue locally preserving the result data, and handle next event information.
If it is determined that the time span at write-in moment to the current time is less than setting time length, real time computation system
It can then continue in the result data that locally preservation step S101 is obtained, and continue to handle next event information.
By the above method, real time computation system is only when the time span that moment to current time is written is more than setting
Between length when just write result data to database, if being less than setting time length, real time computation system can temporarily exist
It is local to preserve result data, and subsequent event information is handled, locate again without waiting until to write result data to after database
Subsequent event information is managed, therefore can be under the premise of ensureing that real time computation system is consistent with the data in database（Postponing
It can ensure that real time computation system is consistent with the data in database after above-mentioned setting time length）, effectively improve event information
Treatment effeciency will not cause the accumulation of event information, reduce the probability to break down due to event information is accumulated.
Further, in due to handling scene in logistics information, it is transferred to the logistics event information tool of real time computation system
There is the characteristics of continuous, data volume is larger, and be not in bulk transfer to real time computation system discrete, in batches, therefore,
Above-mentioned data processing method as shown in Figure 1 can be used in logistics information processing scene, that is, current described in the embodiment of the present application
Event information and next event information include logistics event information, and the result data includes logistics result data.Certainly,
Data processing method as shown in Figure 1, which can also be used for other, to be had the characteristics that in data processing scene continuous, that data volume is larger,
Such as, commodity transaction information processing scene.
Below for applying data processing method as shown in Figure 1 in handling scene in logistics information, to the application reality
The effect for applying the data processing method of example offer illustrates.
Logistics information handle scene in, event information mainly include certain logistics facility deliver event information, pull
The event information of receipts, the event information of entry/exit terminal, event information for signing for etc., and each event information may result in
The update of multiple logistics result datas.
For example, the event information for " logistics facility A performs order 1 delivery operation ", it is assumed that order 1 sells household
Family is user a, and place of departure is city b, and place of acceptance is city c, then the event information can lead to " logistics facility A delivering amounts ", " object
The delivering amount that stream mechanism A is provided for user a ", " delivering amount that logistics facility A delivers from city b ", " logistics facility A is sent to city
The update of this 4 logistics result datas of the delivering amount of c "（The end value carried in this 4 result datas is specifically added 1）.
For another example, for the event information of " logistics facility A performs order 1 and pulls work of bringing drill to an end ", it is assumed that upper one is pulled receipts ground
It is city e with currently pulling receipts for city d, next is city f with pulling receipts, then the event information can lead to that " logistics facility A is pulled
Receipts amount ", " logistics facility A pulls receipts amount in city e from what city d pulled receipts ", " logistics facility A pulls receipts amount city e's ", " logistics
Mechanism A is pulled from city e receive after be sent to the delivering amount of city f " updates of this 4 logistics result datas（Specifically by this 4 results
The end value carried in data adds 1）.
It will again be assumed that the number maximum per second that data can be written to database is 10000 times, average each logistics event letter
Breath can lead to the update of 4 logistics result datas, since logistics event information is not that discrete bulk transfer is calculated to real-time
System, but successively and be largely transferred to real time computation system, it is assumed that it is per second to be transferred in real time computation system
Logistics event information is 20000, then according to the method for the prior art, treatment progress for above-mentioned logistics event information,
Need after updated 4 logistics result datas are written database, could handle next logistics event information, it is per second most
10000/4=2500 logistics event information can be mostly handled, far below the speed per second for receiving 20000 logistics event informations
Degree, therefore event information can be caused to accumulate.And according to the application method as shown in Figure 1, if predetermined time period is 10 seconds,
Then since a logistics result just only is written to database when the time span that moment to current time is written is more than 10 seconds
Data can directly handle next logistics event information if being less than, therefore real time computation system is per second can at most handle
10000/4 × 10=25000 event informations, more than the speed per second for receiving 20000 logistics event informations, therefore, in object
In stream information processing scene, data processing method provided by the embodiments of the present application compared with the prior art under middle the same terms in real time
For the computing system efficiency per second that can at most handle 2500 event informations, the data processing method that the application provides is by event
The treatment effeciency of information improves 10 times, will not cause the accumulation of event information.In addition, due to handling scene for logistics information
For, the logistics result data obtained after update is postponed 10 seconds write-in databases can meet logistics business data query
It is required that, therefore, the logistics result data obtained after update is postponed 10 seconds write-in databases will not be to logistics business data
Inquiry cause large effect.
Further, due in the embodiment of the present application real time computation system be locally preserve result data.Therefore, exist
In step S101 shown in FIG. 1, real time computation system in processing current event information and during obtaining result data, according to
Current event information, to the carrying attribute information locally preserved（The attribute information is the attribute carried in current event information
Information）When the result data of corresponding Data Identification is updated, the carrying attribute information locally preserved can be specifically extracted
The result data of corresponding Data Identification, and the result data of extraction is updated, if not number from be locally extracted to carrying
According to the result data of mark, then the data for carrying the Data Identification can be read from database to local, and to reading this
The data on ground are updated.In addition, if real time computation system does not also read from database the data for carrying the Data Identification,
It then can be in local（Such as local memory or hard disk）Addition carries the result data of the Data Identification, and is believed according to the current event
Breath, is updated the result data of addition, alternatively, can also be set to the end value carried in the result data of addition pre-
If initial value, and according to the current event information, the result carried in the result data of addition is only updated.Wherein,
The preset initial value can be set, such as be set as 0 according to actual needs.
For example, treatment progress 1 determines that the attribute information carried in current event information " logistics facility A+ deliveries operation " is right
The Data Identification answered is " logistics machine for the carrying Data Identification after " logistics facility A delivering amounts ", preserved in extractable local memory
The result data of structure A delivering amounts ", and the result data of extraction is updated.If carrying is not extracted from local memory
Data Identification is the result data of " logistics facility A delivering amounts ", that is, there is no carry Data Identification as " logistics machine in memory
The result data of structure A delivering amounts ", then from database read carry Data Identification for " logistics facility A delivering amounts " data to
Local memory, and the result data of reading is updated.If it is " logistics not read from database and carry Data Identification
The data of mechanism A delivering amounts ", that is, there is no carry Data Identification as " logistics facility A in local memory and in database
The data of delivering amount " can then add the result data for carrying that Data Identification is " logistics facility A delivering amounts " in local memory,
And the end value carried in the result data of addition is set to 0, and the result data of addition is updated.
The above method can ensure some result data of real time computation system local in update, by predetermined time period
After be written in database, that is, the data in database after predetermined time period can with real time computation system in
Accordingly result data in depositing are consistent, and in other words, real time computation system is having updated a result data and passing through default
After time span, you can the data are correctly being inquired from database.However, can exist in practical application scene real-time
There is situation that is abnormal and interrupting in some treatment progress in computing system, if some treatment progress interrupts, then this handle into
Journey is stored in local result data and can also be emptied, and again by result after the delay predetermined time period for using the application above-mentioned
The method that database is written in data if local result data is not yet written into database before being emptied, will appear
The situation that local result data is lost and causes database inconsistent with the data in real time computation system, this may result in not
Correct data can be inquired in the database, can reduce the accuracy of data processing.
Therefore, in order to ensure the accuracy of the data stored in database, treatment progress is passing through in the embodiment of the present application
After step S101 processing current event information obtains result data and is stored in local, also obtained result data is recorded in
In journal file.Can also monitoring device be set in real time computation system or except the real time computation system, for real-time
Each treatment progress of computing system is monitored.It is when monitoring treatment progress exception, then corresponding every for the treatment progress
The result data for carrying the Data Identification that the treatment progress records for the last time in journal file is written a Data Identification
Into database.In this way, it can still ensure data and meter in real time in database when treatment progress exception occurs and interrupts
The data of calculation system local are consistent, can effectively improve the accuracy of the data stored in database.Wherein, treatment progress will obtain
Result data when being recorded in journal file, the temporal information for obtaining result data can be recorded in journal file, then it is real
When computing system when being write result data in database according to journal file, can be directed to each Data Identification, this is handled
Carrying Data Identification and the result data of corresponding temporal information the latest of process record are written in database.
Wherein, above-mentioned monitoring device can be ZooKeeper components, which may be disposed at real-time calculating
On the calculation server of system, it may also set up in the other systems independently of real time computation system.For real time computation system
A treatment progress for, the treatment progress is in normal operation, even if not handling any event information, the treatment progress
It is in operating status rather than interrupt status, therefore, according to ZooKeeper monitor component treatment progress, treatment progress can
A transient node is registered on ZooKeeper components on startup, the transient node is only corresponding with the treatment progress, if should
Treatment progress is in operating status, then the transient node always exists, once the treatment progress interrupts, then the transient node just disappears
It loses, so as to which ZooKeeper components can be monitored each transient node of itself, once find that some transient node disappears,
It can determine that the corresponding treatment progress of the transient node occurs abnormal and interrupts, so as to notify real time computation system according to daily record
Record in file, for the corresponding each Data Identification of the treatment progress, by the treatment progress in journal file last
The result data of the carrying of the secondary record Data Identification is written in database, to ensure the accuracy of data in database.
Illustrate by taking Fig. 2 and Fig. 3 as an example below in the embodiment of the present application under normal circumstances with the data processing under abnormal conditions
Method.
Assuming that real time computation system has 3 treatment progress, respectively process 1, process 2 and process 3, this 3 treatment progress
Corresponding Data Identification is respectively R1, R2, R3, and result data is all stored in local memory by each treatment progress.
Assuming that real time computation system has received two current event information, respectively event 1 and event 3, event 1 carries
Attribute information correspond to process 1 and Data Identification R1, the attribute information that event 3 carries corresponds to process 3 and Data Identification R3.Currently
Two end values are carried in the result data of carrying R1 preserved in memory, the first end value is 100, and the second end value is 200,
Also two end values are carried in the result data of carrying R3 preserved in current memory, the first end value is 300, the second end value
It is 400.The R1 corresponding write-in moment is 11 days 12 November in 2011:00:00, R3 corresponding write-in moment was November 11 in 2011
Day 12:01:00, current time is 11 days 12 November in 2011:01:02.Predetermined time period is 10 seconds.Then：
Fig. 2 is the schematic diagram of data handling procedure under normal circumstances provided by the embodiments of the present application, in fig. 2, at process 1
The first end value carried in memory in the result data that Data Identification is R1 is then updated to 101, and will by director's part 1 by 100
Second end value is updated to 201 by 200.Process 3 handles event 3, then the result data that Data Identification is R3 will be carried in memory
In the first end value be updated to 301, and the second end value is updated to 401 by 400 by 300.Process 2 does not handle event.
For process 1, obtain carrying the result data of Data Identification R1, and due to the R1 corresponding write-in moment
For 11 days 12 November in 2011:00:00, current time is 11 days 12 November in 2011:01:02, therefore during the corresponding write-ins of R1
The time span for being carved into current time has been more than predetermined time period 10 seconds, so as to which process 1 will carry Data Identification R1 in memory
Result data be written in database, and update the R1 preserved in memory it is corresponding write-in the moment be 2011 years 11 current time
On the moon 11 12:01:02.
For process 3, obtain carrying the result data of Data Identification R3, and due to the R3 corresponding write-in moment
For 11 days 12 November in 2011:01:00, current time is 11 days 12 November in 2011:01:02, therefore during the corresponding write-ins of R1
The time span for being carved into current time is less than predetermined time period 10 seconds, so as to which process 3 wouldn't will carry data mark in memory
The result data for knowing R3 is written in database, can handle itself next event information to be dealt with.
Certainly, process 1 and process 3 are also by obtained result data and the temporal information for obtaining result data（It obtains
The temporal information of result data is 11 days 12 November of 2011 current time:01:02）It is recorded in journal file.
In process shown in Fig. 2, ZooKeeper components are monitored process 1, process 2 and process 3, to this 3 into
Cheng Jun does not monitor exception, therefore real time computation system need not write result data to database according to journal file, moreover, taking
Result data with Data Identification R1 is consistent with being to maintain in the memory of real time computation system in the database, still, due to
The time span at R3 corresponding write-in moment to current times is less than 10 seconds, therefore the number of Data Identification R3 is carried in database
It is not consistent according to the result data that Data Identification R3 is carried in the memory with real time computation system.
Fig. 3 is the schematic diagram of data handling procedure under abnormal conditions provided by the embodiments of the present application, in fig. 3, it is assumed that
11 days 12 November in 2011:02:When 00, process 3 occurs abnormal and interrupts, then ZooKeeper monitor components to process 3 occur different
Often, therefore notice real time computation system is for the corresponding each Data Identification of process 3, determines that process 3 is last in journal file
What is once recorded carries the result data of the Data Identification.
Real time computation system determines that the result data of carrying R3 that process 3 records for the last time in journal file is 2011
On November 11,12 in:01:The result data of 02 record, therefore the result data of carrying R3 is written in database, to protect
When card process 3 occurs abnormal, the data stored in database are still accurate.
Certainly, after data being written into database according to journal file, the number that can will also be carried in the result data of write-in
At the time of write-in database being updated to according to the mark corresponding write-in moment.Above-mentioned is only to correspond to a data with process 3 to be identified as
What example illustrated, in fact, process 3 can correspond to multiple Data Identifications.
It should be noted that above-mentioned Fig. 2 and Fig. 3 are the monitoring devices for monitoring treatment progress（Such as Fig. 2 and Fig. 3 institutes
The ZooKeeper components shown）It is deployed in for real time computation system inside and illustrates, monitoring device can also be independently of
Real time computation system is disposed, and is just no longer repeated one by one here.
In addition, in order to further ensure the accuracy of data preserved in database, real time computation system can also meet
During preset trigger condition, all result datas locally preserved are written in database.Wherein, which can
To be：According to the period of setting, when the finish time for determining current period arrives, determine to meet preset trigger condition.This
Be due in the embodiment of the present application for real time computation system locally preserve carrying some Data Identification result data and
Speech, when the result data only locally preserved is updated, can just judge the Data Identification it is corresponding write-in the moment to it is current when
Whether the time span at quarter is more than predetermined time period, so as to decide whether to write updated result data according to judging result
Enter in database, therefore, inevitably there are following extreme cases in practical application scene：
When one result data is updated, it is long to judge that the time span that moment to current time is written is less than preset time
Degree, therefore the result data is not written into database, but the subsequent result data within very long a period of time all not by more
Newly, therefore the result data is caused all to be not written in database in a very long time.
When there is above-mentioned extreme case, the accuracy for also resulting in the data stored in database declines, therefore, in real time
All result datas locally preserved, in each end cycle, can be all written to number by computing system according to the period of setting
According in library, the wherein period of the setting is more than above-mentioned setting time length.For example, real time computation system can incite somebody to action locally for every 24 hours
In all result datas write-in database preserved, lead to the data stored in database to avoid there is above-mentioned extreme case
The problem of inaccurate.
Above-mentioned preset trigger condition can also be：Result data in memory is written to when real time computation system receives
During write instruction in database, determine to meet preset trigger condition, all result datas locally preserved are written to number
According in library.
Further, it may also include：Set specified services type in the database in advance, the database is receiving
To after the inquiry request of user, whether the type of service to be inquired for judging to carry in inquiry request is preset specified services
Type, if so, sending the write instruction being written to the result data of specified services type in database.
Due in practical application scene, if some business need real time computation system cannot after result data is obtained
Postpone the long time writes result data to database again.It, can be pre- to provide a user more accurate, timely result data
Specified services type is first set in the database（For example, the type of service that requirement of real-time is high, such as requires the industry that delay time is short
Service type）, database then can determine whether the business to be inquired carried in inquiry request after the inquiry request of user is received
Whether type is preset specified services type, is referred to if so, sending the write-in being written to all result datas in database
It enables, at this point, can be written to all result datas locally preserved in database, then provide a user data, if not pre-
If specified services type when, can its data inquired directly be provided a user according to database query result.
It is the method for data processing provided by the embodiments of the present application above, based on same thinking, the embodiment of the present application is also
A kind of device of data processing is provided, as shown in Figure 4.
Fig. 4 is the apparatus structure schematic diagram of data processing provided by the embodiments of the present application, is specifically included：
Event processing module 401 for handling current event information, obtains result data, and be stored in local；
Determining module 402 for the Data Identification carried according to the result data, determines the Data Identification of record
The result data for carrying the Data Identification is written in database by corresponding write-in moment, said write moment for the last time
At the time of；
Judgment module 403, whether the time span for judging said write moment to current time is more than setting time
Length；
Writing module 404, for the number of results that when the judging result of the judgment module is to be, will locally preserve
Current time is updated to according to being written in database, and by the Data Identification corresponding write-in moment；
The event processing module 401 is additionally operable to, and when the judging result of the judgment module is no, is continued in local guarantor
The result data is deposited, and next event information is handled.
The current event information and next event information include logistics event information, and the result data includes
Logistics result data.
Attribute information is carried in current event information, attribute information is corresponding with treatment progress and Data Identification；
The event processing module 401 is specifically used for, according to the attribute information carried in the current event information, by institute
It states current event information and distributes to the corresponding treatment progress of the attribute information, make the treatment progress according to the current event
Information is updated the result data of the corresponding Data Identification of the carrying attribute information locally preserved, will be updated
Result data is as the result data handled current event information.
The event processing module 401 is specifically used for, and extracts the corresponding data of the carrying attribute information locally preserved
The result data of mark, and the result data of extraction is updated, if the not Data Identification from be locally extracted to carrying
Result data then reads the data for carrying the Data Identification to local from database, and to read local data into
Row update.
The obtained result data is recorded in journal file by the treatment progress；
Described device further includes：
Monitoring module 405, for being monitored to the treatment progress, when monitoring the treatment progress exception, needle
Each Data Identification corresponding to the treatment progress, the treatment progress is recorded for the last time in the journal file
The result data for carrying the Data Identification is written in database.
Said write module 404 is additionally operable to, and when meeting preset trigger condition, all result datas locally preserved are write
Enter into database, wherein, the preset trigger condition includes：It receives all result datas that will locally preserve and is written to number
According to the write instruction in library.
Further, specified services type is set in the database in advance, said write instruction is, in the data
After library receives the inquiry request of user, whether the type of service to be inquired for judging to carry in inquiry request is preset finger
Determine type of service, if so, sending the write instruction result data of specified services type being written in database.
The device of specific above-mentioned data processing can be located in real time computation system.
The embodiment of the present application provides a kind of method and device of data processing, and this method real time computation system is to current event
After information processing obtains result data, according to the Data Identification that the result data carries, judge that the Data Identification of record corresponds to
Write-in moment to current time time span whether be more than setting time length, if so, the result data is written to
In database, and the Data Identification corresponding write-in moment is updated to current time, otherwise continues locally preserving the result
Data, and handle next event information.The above method is since real time computation system is only at the write-in moment to current time
Time span just writes result data to database when being more than setting time length, remaining time real time computation system can be handled
Subsequent event information, therefore can be effectively improved under the premise of ensureing that real time computation system is consistent with the data in database
The treatment effeciency of event information will not cause the accumulation of event information, can effectively reduce and break down due to event information is accumulated
Probability.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net
Network interface and memory.
Memory may include computer-readable medium in volatile memory, random access memory (RAM) and/or
The forms such as Nonvolatile memory, such as read-only memory (ROM) or flash memory (flashRAM).Memory is showing for computer-readable medium
Example.
Computer-readable medium includes permanent and non-permanent, removable and non-removable media can be by any method
Or technology come realize information store.Information can be computer-readable instruction, data structure, the module of program or other data.
The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), moves
State random access memory (DRAM), other kinds of random access memory (RAM), read-only memory (ROM), electric erasable
Programmable read only memory (EEPROM), fast flash memory bank or other memory techniques, CD-ROM read-only memory (CD-ROM),
Digital versatile disc (DVD) or other optical storages, magnetic tape cassette, the storage of tape magnetic rigid disk or other magnetic storage apparatus
Or any other non-transmission medium, available for storing the information that can be accessed by a computing device.It defines, calculates according to herein
Machine readable medium does not include temporary computer readable media (transitorymedia), such as data-signal and carrier wave of modulation.
It should also be noted that, term " comprising ", "comprising" or its any other variant are intended to nonexcludability
Comprising so that process, method, commodity or equipment including a series of elements are not only including those elements, but also wrap
Include other elements that are not explicitly listed or further include for this process, method, commodity or equipment it is intrinsic will
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that wanted including described
Also there are other identical elements in the process of element, method, commodity or equipment.
It will be understood by those skilled in the art that embodiments herein can be provided as method, system or computer program product.
Therefore, complete hardware embodiment, complete software embodiment or the embodiment in terms of combining software and hardware can be used in the application
Form.It is deposited moreover, the application can be used to can be used in one or more computers for wherein including computer usable program code
Storage media（Including but not limited to magnetic disk storage, CD-ROM, optical memory etc.）The shape of the computer program product of upper implementation
Formula.
The foregoing is merely embodiments herein, are not limited to the application.For those skilled in the art
For, the application can have various modifications and variations.All any modifications made within spirit herein and principle are equal
Replace, improve etc., it should be included within the scope of claims hereof.
Claims (12)
- A kind of 1. method of data processing, which is characterized in that including：Current event information is handled, obtains result data, and be stored in local；According to the Data Identification that the result data carries, the Data Identification corresponding write-in moment of record is determined, it is described At the time of the result data for carrying the Data Identification is written in database by the write-in moment for the last time；Whether the time span for judging said write moment to current time is more than setting time length；If so, the result data locally preserved is written in database, and by the corresponding write-in of the Data Identification Moment is updated to current time；Otherwise, continue locally preserving the result data, and handle next event information.
- 2. the method as described in claim 1, which is characterized in that the current event information and next event information packet Logistics event information is included, the result data includes logistics result data.
- 3. the method as described in claim 1, which is characterized in that carry attribute information in current event information, attribute information with Treatment progress and Data Identification correspond to；Current event information is handled, obtains result data, is specifically included：According to the attribute information carried in the current event information, the current event information is distributed into the attribute information Corresponding treatment progress；The treatment progress is according to the current event information, data mark corresponding to the carrying locally preserved the attribute information The result data of knowledge is updated, using updated result data as the number of results handled current event information According to.
- 4. method as claimed in claim 3, which is characterized in that data corresponding to the carrying locally preserved the attribute information The result data of mark is updated, and is specifically included：The result data of the corresponding Data Identification of the carrying attribute information locally preserved is extracted, and to the result data of extraction It is updated；If the result data of the Data Identification not from be locally extracted to carrying, read from database and carry the data mark The data of knowledge are updated to local to reading local data.
- 5. method as claimed in claim 3, which is characterized in that the method further includes：The obtained result data is recorded in journal file by the treatment progress；The treatment progress is monitored；When monitoring the treatment progress exception, for the corresponding each Data Identification of the treatment progress, by the processing The result data for the carrying Data Identification that process records for the last time in the journal file is written in database.
- 6. the method as described in claim 1, which is characterized in that the method further includes：When meeting preset trigger condition, all result datas locally preserved are written in database；Wherein, the preset trigger condition includes：Receive the write-in that the result data that will locally preserve is written in database Instruction.
- 7. method as claimed in claim 6, which is characterized in that the method further includes：Set specified services type in the database in advance, the database is sentenced after the inquiry request of user is received Whether the type of service to be inquired carried in disconnected inquiry request is preset specified services type, if so, sending will refer to The result data for determining type of service is written to write instruction in database.
- 8. a kind of device of data processing, which is characterized in that including：Event processing module for handling current event information, obtains result data, and be stored in local；Determining module for the Data Identification carried according to the result data, determines that the Data Identification of record is corresponding Be written the moment, the said write moment for it is last by the result data for carrying the Data Identification be written in database when It carves；Judgment module, whether the time span for judging said write moment to current time is more than setting time length；Writing module, for when the result data locally preserved is when being, is written by the judging result of the judgment module Current time is updated into database, and by the Data Identification corresponding write-in moment；The event processing module is additionally operable to, and when the judging result of the judgment module is no, is continued described in locally preservation Result data, and next event information is handled.
- 9. device as claimed in claim 8, which is characterized in that the current event information and next event information packet Logistics event information is included, the result data includes logistics result data.
- 10. device as claimed in claim 8, which is characterized in that carry attribute information in current event information, attribute information with Treatment progress and Data Identification correspond to；The event processing module is specifically used for, will be described current according to the attribute information carried in the current event information Event information distributes to the corresponding treatment progress of the attribute information, makes the treatment progress according to the current event information, The result data of the corresponding Data Identification of the carrying attribute information locally preserved is updated, by updated number of results According to as the result data handled current event information.
- 11. device as claimed in claim 10, which is characterized in that the event processing module is specifically used for, and extracts local protect The result data of the corresponding Data Identification of the carrying attribute information deposited, and the result data of extraction is updated, if not The result data of the Data Identification from be locally extracted to carrying then reads the data for carrying the Data Identification from database It is updated to local, and to reading local data.
- 12. device as claimed in claim 11, which is characterized in that the treatment progress records the obtained result data In journal file；Described device further includes：Monitoring module, for being monitored to the treatment progress, when monitoring the treatment progress exception, for the place The corresponding each Data Identification of reason process, the carrying that the treatment progress is recorded for the last time in the journal file number It is written in database according to the result data of mark.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410053223.4A CN104850556B (en) | 2014-02-17 | 2014-02-17 | A kind of method and device of data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410053223.4A CN104850556B (en) | 2014-02-17 | 2014-02-17 | A kind of method and device of data processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104850556A CN104850556A (en) | 2015-08-19 |
| CN104850556B true CN104850556B (en) | 2018-06-29 |
Family
ID=53850203
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410053223.4A Active CN104850556B (en) | 2014-02-17 | 2014-02-17 | A kind of method and device of data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104850556B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203531A (en) * | 2016-03-16 | 2017-09-26 | 阿里巴巴集团控股有限公司 | A kind of document handling method and device |
| CN107784021A (en) * | 2016-08-31 | 2018-03-09 | 北京国双科技有限公司 | The method, apparatus and system that control data is deleted |
| CN110460902B (en) * | 2018-05-08 | 2022-02-22 | 腾讯科技（深圳）有限公司 | Media information playing method and device, storage medium and electronic device |
| CN109709587B (en) * | 2018-12-27 | 2022-03-11 | 上海司南卫星导航技术股份有限公司 | Multi-event processing method and circuit thereof |
| CN113377792A (en) * | 2021-06-10 | 2021-09-10 | 上海微盟企业发展有限公司 | Data write-back method and device, electronic equipment and storage medium |
| CN113177032B (en) * | 2021-06-29 | 2021-10-22 | 南京云联数科科技有限公司 | Database-based data sharing method and system |
| CN113780017B (en) * | 2021-09-03 | 2024-02-09 | 珠海格力电器股份有限公司 | Near field communication triggering method, device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102075670A (en) * | 2009-11-24 | 2011-05-25 | 新奥特（北京）视频技术有限公司 | Log recording method and device for broadcast machine |
| CN102609337A (en) * | 2012-01-19 | 2012-07-25 | 北京神州数码思特奇信息技术股份有限公司 | Rapid data recovery method for memory database |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100174863A1 (en) * | 2007-11-30 | 2010-07-08 | Yahoo! Inc. | System for providing scalable in-memory caching for a distributed database |
-
2014
- 2014-02-17 CN CN201410053223.4A patent/CN104850556B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102075670A (en) * | 2009-11-24 | 2011-05-25 | 新奥特（北京）视频技术有限公司 | Log recording method and device for broadcast machine |
| CN102810050A (en) * | 2011-05-31 | 2012-12-05 | 深圳市金蝶友商电子商务服务有限公司 | Log data writing method and log system |
| CN102609337A (en) * | 2012-01-19 | 2012-07-25 | 北京神州数码思特奇信息技术股份有限公司 | Rapid data recovery method for memory database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104850556A (en) | 2015-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104850556B (en) | A kind of method and device of data processing | |
| CN108615119B (en) | Abnormal user identification method and equipment | |
| CN107395665A (en) | A kind of block chain service handling and business common recognition method and device | |
| CN106997431B (en) | Data processing method and device | |
| CN108074164B (en) | Order processing method and device | |
| CN106202280B (en) | Information processing method and server | |
| US8422786B2 (en) | Analyzing documents using stored templates | |
| CN108681866B (en) | Waybill processing method, system, device and storage medium | |
| CN103581626A (en) | Video monitoring system and video storage information recording method | |
| CN106815254A (en) | A kind of data processing method and device | |
| CN109756760A (en) | Generation method, device and the server of video tab | |
| US20230205755A1 (en) | Methods and systems for improved search for data loss prevention | |
| CN108255628A (en) | A kind of data processing method and device | |
| CN111090705B (en) | Multidimensional data processing method, device and equipment and storage medium | |
| CN111400056A (en) | Message queue-based message transmission method, device and equipment | |
| CN108134812A (en) | Data processing method and device | |
| CN106570005A (en) | Database cleaning method and device | |
| CN109597566A (en) | A kind of reading data, storage method and device | |
| CN111552575B (en) | Message consumption method, device and equipment based on message queue | |
| CN105988881B (en) | Method and device for processing resource access operation information | |
| CN107295059A (en) | The statistical system and method for service propelling amount | |
| CN111078588B (en) | Garbage recycling method, device, equipment and storage medium | |
| CN108537577B (en) | Data validity query method and device, storage medium and server | |
| CN112948501B (en) | Data analysis method, device and system | |
| CN109003643A (en) | A kind of data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| EXSB | Decision made by sipo to initiate substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |