CN110750547A - Data acquisition method and device - Google Patents
Data acquisition method and device Download PDFInfo
- Publication number
- CN110750547A CN110750547A CN201911002722.XA CN201911002722A CN110750547A CN 110750547 A CN110750547 A CN 110750547A CN 201911002722 A CN201911002722 A CN 201911002722A CN 110750547 A CN110750547 A CN 110750547A
- Authority
- CN
- China
- Prior art keywords
- data
- version
- data version
- currently
- consumer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data acquisition method and a data acquisition device, which are applied to a data consumer, wherein a data version acquisition request is sent to the data producer to obtain a data version returned by the data producer according to the data version acquisition request, the obtained data version is compared with a consumed data version in the data consumer to obtain a current missing data version of the data consumer, and data corresponding to the current missing data version is obtained from the data producer.
Description
Technical Field
The present invention relates to the field of data acquisition, and in particular, to a data acquisition method and apparatus.
Background
With the development of internet technology, the importance of data acquisition is increasing day by day.
During data acquisition, data consumers and data producers play an important role. The data consumer is the party receiving and using the data, and the data producer is the party generating and outputting the data.
The data consumer may obtain data from the data producer in an active obtaining process, where the active obtaining process is that the data consumer sends a data request to the data producer, and then the data consumer receives data of a new data version sent by the data producer. Normally, a data consumer can obtain the updated data version of data sent by the data producer.
However, some reasons (e.g., the data producer cannot send data) may result in the data consumer not receiving the data sent by the data producer for a certain period of time, which is a data transmission interruption period. If the data producer keeps data evolution and updates the data version during the data transmission interruption period, when the data consumer receives the data sent by the data producer again, the data consumer may lack the data evolved in the data version updated by the data producer during the data transmission interruption period.
Disclosure of Invention
In view of the above problems, the present invention provides a data acquisition method and apparatus that overcomes or at least partially solves the above problems, and the technical solution is as follows:
a data acquisition method for use by a data consumer, the method comprising:
sending a data version acquisition request to a data producer;
acquiring a data version returned by the data producer according to the data version acquisition request;
comparing the obtained data version with the consumed data version in the data consumer to obtain the data version which is currently lost by the data consumer;
obtaining data corresponding to the currently missing data version from the data producer.
Optionally, the sending a data version obtaining request to a data producer includes:
and sending a data version acquisition request to a data producer according to a preset period.
Optionally, the data version obtaining request is used to request to obtain all data versions at the data producer.
Optionally, the comparing the obtained data version with the data version already consumed by the data consumer to obtain the data version currently missing by the data consumer includes:
determining a data version subsequent to and adjacent to the currently consumed data version among the obtained data versions as a first data version, and determining a latest data version at the data producer as a second data version;
determining the first data version to the second data version as the data version currently missing by the data consumer.
Optionally, the comparing the obtained data version with the data version already consumed by the data consumer to obtain the data version currently missing by the data consumer includes:
determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a currently effective data version at the data producer as a third data version;
determining the first to third data versions as data versions currently missing by the data consumer.
Optionally, after obtaining the data corresponding to the currently missing data version from the data producer, the method further includes:
and according to the time sequence of each data version in the current missing data version, sequentially using the data corresponding to the current missing data version to perform updating processing.
Optionally, the sequentially using the data corresponding to the currently missing data version to perform the update processing according to the time sequence of each data version in the currently missing data version includes:
taking a data version which is later than the currently consumed data version and is closest to the currently consumed data version as a current data version to be updated in the currently missing data versions;
updating the data corresponding to the currently consumed data version by using the data corresponding to the currently to-be-updated data version;
and determining the current data version to be updated as the current consumed data version, returning to execute the step of taking the data version which is later than the current consumed data version and is closest to the current consumed data version as the current data version to be updated in the current missing data version.
A data acquisition apparatus comprising an acquisition request transmission unit, a first data version acquisition unit, a second data version acquisition unit, and a data acquisition unit, wherein:
the acquisition request sending unit is used for sending a data version acquisition request to a data producer;
the first data version obtaining unit is used for obtaining the data version returned by the data producer according to the data version obtaining request;
the second data version obtaining unit is used for comparing the obtained data version with the consumed data version in the data consumer to obtain the data version which is currently lost by the data consumer;
the data obtaining unit is used for obtaining data corresponding to the current missing data version from the data producer.
Optionally, the obtaining request sending unit is specifically configured to send the data version obtaining request to the data producer according to a preset period.
Optionally, the data version obtaining request is used to request to obtain all data versions at the data producer.
Optionally, the second data version obtaining unit specifically includes a first data version determining unit and a second data version determining unit, where:
the first data version determining unit is used for determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version and determining a latest data version at the data producer as a second data version;
the second data version determining unit is used for determining the first data version to the second data version as the data versions which are currently missing by the data consumer.
Optionally, the second data version obtaining unit specifically includes a third data version determining unit and a fourth data version determining unit, where:
the third data version determining unit is used for determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a currently effective data version at the data producer as a third data version;
the fourth data version determining unit is configured to determine the first to third data versions as data versions currently missing by the data consumer.
Optionally, the data obtaining apparatus further includes a first data updating unit, where the first data updating unit is configured to, after obtaining the data corresponding to the currently missing data version from the data producer, sequentially use the data corresponding to the currently missing data version for updating according to a time sequence of each data version in the currently missing data version.
Optionally, the first data updating unit specifically includes: a fifth data version determination unit, a second data update unit, and a return execution unit, wherein:
the fifth data version determining unit is used for taking the data version which is later than the currently consumed data version and is closest to the currently consumed data version as the current data version to be updated in the currently missing data versions;
the second data updating unit is used for updating the data corresponding to the currently consumed data version by using the data corresponding to the currently to-be-updated data version;
and the return execution unit is used for determining the current data version to be updated as the current consumed data version, and returning to execute the step of taking the data version which is later than the current consumed data version and is closest to the current consumed data version as the current data version to be updated in the current missing data version.
According to the data acquisition method and device disclosed by the invention, the data version which is less than the data version which is currently consumed by the data consumer is sent to the data producer, the currently missing data version of the data consumer is determined according to the obtained data version, and then the data required by the data consumer is acquired according to the currently missing data version of the data consumer, so that when the data consumer and the data producer have a data transmission interruption period, the data consumer can pertinently acquire the data which can be evolved by the data producer in the data transmission interruption period in the subsequent normal data transmission process, the missing of the data evolved by the data producer by the data consumer is avoided, and the data transmission quantity and the pressure on the data broadband in the data transmission process are also reduced.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
fig. 1 is a flowchart illustrating a data acquisition method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data acquisition method provided by an embodiment of the invention;
FIG. 3 is a flow chart of a data acquisition method according to an embodiment of the present invention;
FIG. 4 is a flow chart of another data acquisition method provided by embodiments of the present invention;
FIG. 5 is a flow chart of a data acquisition method provided by an embodiment of the invention;
fig. 6 is a schematic structural diagram illustrating a data acquisition apparatus according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of another data acquisition apparatus provided in an embodiment of the present invention;
fig. 8 is a schematic structural diagram illustrating a data acquisition apparatus according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As shown in fig. 1, an embodiment of the present invention provides a data acquisition method applied to a data consumer, where the method may include the following steps:
s100, sending a data version acquisition request to a data producer;
the data consumer is the party receiving and using the data, and the data producer is the party generating and outputting the data. In practical applications, the data consumer may be a device such as a mobile phone, an ipad, or a digital tv set-top box, and the data producer may be a device such as a sensor, a server, or a chip.
The data version obtaining request is a request for obtaining the marks of the data versions, and each mark of the data version can uniquely identify one data version. It should be noted that, the present invention is a mark for obtaining the data version when obtaining the data version.
Alternatively, the present invention may use numerical numbering to label different versions of data. For example, the flag of the current latest data version is set to 19.0.0.3, and the flag of the data version and the data corresponding to the data version are retained before the data corresponding to the data version is subjected to data evolution. When data evolution is carried out, a new data version with the number of 19.0.0.4 (namely, adding 1 to the number at the end of 19.0.0.3, when the number at a certain position (the numbers at adjacent positions are separated by dot symbols) is added to 10, the number at the position becomes 0, and the number at the position before the position is added to 1) is created, and after the data evolution is completed, the data with the completed evolution is associated with the new data version with the number of 19.0.0.4, and the new data version with the number of 19.0.0.4 is taken as the current latest data version.
The invention may also mark different data versions with numbers representing time of generation in the digital format "yyymmddhhmmss" where, from left to right, yyyy represents year, mm represents month, dd represents day, hh represents clock, mm represents minute, ss represents seconds, for example: the label of a data version is 20190501123050, which means that the data version is generated in 2019, 5, 1, 12, 30 minutes and 50 seconds. It should be noted that the manner in which the version of the data is marked is not limited in the present invention.
It should be noted that, when data corresponding to a certain data version (hereinafter referred to as an old data version) evolves (i.e. the data is being manipulated, such as modified, deleted, and added), a data producer will create a new data version, regardless of the marking rule of each data version as a whole, the new version of data will take effect at a time (after a version of data takes effect, the data consumer can consume the version of data) that is later than the time that the old version of data takes effect, before the effective time of the next non-created data version, therefore, the data versions can be arranged according to the effective time, the new data version can be called as the data before the evolution of the current time, i.e., the next or subsequent data version of the old data version, and the data before the evolution of this time (i.e., the old data version) may be referred to as the previous or previous data version of the new data version.
It should be noted that the data version obtaining request set in step S100 of the present invention should enable the data consumer to obtain at least the data version after the currently consumed data version. For example, the present invention may send a data version acquisition request according to the generation time of the data version currently consumed by the data consumer, i.e., send a request to the data producer to acquire the data version after the generation time of the currently consumed data version, so that the data consumer may acquire the data version after the currently consumed data version. Therefore, the invention can ensure that the data consumer can not always lack the data version and the corresponding data evolved by the data producer.
Optionally, the data corresponding to the new data version may be all data of the data corresponding to the previous data version after the data evolution is completed, or may only include partial data (for example, incremental data) of the all data that is different from the data corresponding to the previous data version, or of course, may also be partial data of the all data that participates in the data evolution. It should be noted that, after receiving the data corresponding to the new data version, the data consumer may complete the update of the data version and the related data according to the received data, no matter how the relationship between the data and all the data is (the update of the data version is the update of the mark of the data version).
Optionally, the sending a data version obtaining request to a data producer includes:
and sending a data version acquisition request to a data producer according to a preset period.
In practical application, the invention can adopt a Quartz scheduling mode to schedule the work of a data consumer (including sending a data version acquisition request, receiving a data version, receiving data and the like), not only can realize that the data consumer sends the data version acquisition request to a data producer at regular time, but also can ensure that the data consumer does not depend on the data producer excessively (when a certain data producer does not exist any more, another data producer can be appointed to retransmit data for the data consumer, and the data consumer can start from the data version before data transmission interruption and catch up the data missing in the data transmission interruption period).
The preset period may be set by a technician according to actual conditions, such as an update speed of the data version by the data producer, a demand of the data consumer for a new data version, and the like, which is not limited by the present invention. It should be noted that the data consumer may also send the data version obtaining request to the data producer aperiodically, and the present invention does not limit whether the data consumer sends the data version obtaining request to the data producer periodically.
S200, acquiring a data version returned by the data producer according to the data version acquisition request;
it should be noted that, in step S100, the present invention may include the mark of the latest data version currently owned by the data consumer in the information of the data version acquisition request, so that the data producer can determine the data version currently consumed by the data consumer according to the mark in the information, and further send the data version at least after the data version currently consumed by the data consumer to the data consumer.
For example, the data version currently consumed by the data consumer is labeled 20190501123050 (a number representing the time of generation), then the present invention can carry 20190501123050 in the data version acquisition request that the data consumer sends to the data producer.
In practical applications, the present invention may provide at least one interface at the data producer that can receive a data version obtaining request sent by the data consumer. Through the interface, the data producer can obtain the data version acquisition request of the data consumer and the information in the acquisition request, and can return the mark of the corresponding data version to the data consumer according to the information.
The method and the system can return the mark of the corresponding data version to the data consumer according to the list of the mark of the data version and/or the format of the object Notation (JSON).
For example, when the data version obtaining request is specifically used for obtaining the data versions with the generation time within the time interval 2019.5.100:00: 00-2019.5.1223: 59:59 (the data version currently consumed by the data consumer is marked as 20190501120000, and the mark is already included in the time interval), the data producer will return the mark of the corresponding data version to the data consumer according to the time interval, wherein the list of the marks of all the data versions within the time interval is as follows:
| year, month and day | Time minute second |
| 20190501 | |
| 000000 | |
| 000100 | |
| 000101 | |
| ...... | |
| 20190502 | |
| ...... | |
| ...... | |
| 20190512 | |
| 000000 | |
| 000202 | |
| ...... | |
| 230000 | |
| 232323 |
The tag JSON format for all data versions in this time interval is as follows:
{
“20190501”：[“000000”,”000100”,”000101”...],
“20190502”：[...],
......,
......,
“20190512”：[“000000”,”000202”,...,”230000”,”232323”]
}
when the mark of the data version expressed by the list and/or the JSON format is received, the mark can be identified and stored in the database, and the reading or writing operation can be carried out on the database when the mark is needed.
S300, comparing the obtained data version with the consumed data version in the data consumer to obtain the current missing data version of the data consumer;
wherein the data versions currently missing by the data consumer are data versions that the data consumer lags behind the data producer, which may include data versions that have been validated at the data producer and data versions that have not been validated. It should be noted that the data consumer may consume the data corresponding to the validated data version, and cannot consume the data corresponding to the data version that has not been validated yet.
It should be noted that, in the present invention, the data version currently missing by the data consumer is screened from the obtained data versions, so that the data transmission amount returned by the data producer to the data consumer in step S400 can be reduced, and further, the pressure of the data transmission broadband is reduced.
In practice, the method can compare the obtained marks of each data version with the marks of the consumed data versions in the data consumer respectively to determine whether each data version obtained by the method is consumed by the data consumer, wherein if the obtained mark of a certain data version is the mark of the consumed data version, the obtained data version is not the data version which is currently lost by the data consumer; if not, the obtained data version is the data version which is currently missing by the data consumer.
Wherein, for a data consumer using the database to store the mark of the data version, the invention can lock the mark of the obtained data version by using a data index mode. And then, with the data version currently consumed by the data consumer as a reference, performing traversal comparison in the obtained data versions, and determining whether each data version is a data version consumed by the data consumer. For example, when the number of the generation time is used as the mark of the data version, the number of the generation time of the data version currently consumed by the consumer may be used as a reference number, the number of the generation time of each data version obtained by the present invention is respectively compared with the reference number in size, and the data version corresponding to the number greater than the reference number is determined as the data version missing by the data consumer at the present time, and if the mark of the data version currently consumed by the consumer is 20190501123050, the data versions corresponding to the number greater than 20190501123050 are all the data versions missing by the data consumer at the present time.
The invention can delete the mark which does not belong to the data version which is lost by the data consumer in the current period from the obtained data version when comparing, and only leave the data version which is currently lost by the data consumer.
Alternatively, in the data acquiring method according to another embodiment of the present invention, for the data version acquiring request sent by the data consumer in step S100, the present invention may set it as a request for acquiring all data versions at the data producer, and replace step S300 with steps S310 and S320, as shown in fig. 2:
s310, determining a data version which is next to the currently consumed data version and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a latest data version at the data producer as a second data version;
s320, determining the first data version to the second data version as the data version which is currently missing by the data consumer.
Thus, the data versions obtained in step S310 by the present invention are all data versions at the data producer. Wherein all data versions at the data producer may include data versions currently consumed by the data consumer, historical data versions already consumed by the data consumer, data versions not consumed and validated by the data consumer, and generated data versions not yet validated.
The invention determines the data version which is currently missing by the data consumer from all the data versions obtained by the data producer, can ensure that the number of the currently missing data versions obtained by the data consumer is complete, and further ensures that the data consumer can not miss the data version and the data evolved by the data producer.
S400, obtaining data corresponding to the current missing data version from the data producer.
In the step, the invention can firstly send the data acquisition request containing the mark of the current missing data version to the data producer, and the data producer sends the data of the current missing data version corresponding to the data consumer after receiving the data acquisition request. For example, the data version currently missing by the data consumer is marked with a number interval greater than 20190501123050 and less than or equal to 20190507123049 (the data version is marked with a number indicating the generation time), and after receiving the marked number interval, sent by the data consumer, indicating the data version currently missing by the data consumer, the data producer correspondingly returns the data corresponding to each number in the number interval to the data consumer.
Optionally, the present invention may be provided with at least one data request and send interface at the data producer, that is, the data consumer may send the data acquisition request to the data producer through the interface, and the data producer may send the data to the data consumer through the interface. The data producer may also set at least one data request interface and at least one data transmission interface, respectively.
It should be noted that, in the implementation process of the present invention, two acquisition requests are sent to the data producer together, where the first acquisition request is about the mark of the data version, and the second acquisition request is about the data version. The invention can send the acquisition request only about the data version to the data producer when determining which data versions can be received by the data consumer, without paying attention to the huge data quantity and data details of the data corresponding to each data version, which can reserve the idle computing resources owned by the data producer, and can also lead the data producer to return the data version to the data consumer quickly after receiving the first acquisition request.
It should be noted that, when sending a data version to a data producer that requires to obtain at least a data version that is currently consumed by a data consumer, even if a data transmission interruption period occurs (the data consumer cannot receive data sent by the data producer), the data consumer may send a data version obtaining request for obtaining at least the data version that is currently consumed after the data transmission is recovered to normal, compare the markers of the data versions to determine the currently missing data version, obtain data corresponding to the currently missing data version, and avoid data that the data consumer loses data that the data producer evolved during the data transmission interruption period.
According to the data acquisition method provided by the embodiment, the data version which is less than the data version consumed by the data consumer is sent to the data producer in advance, the data version which is missing at present of the data consumer is determined according to the obtained data version, and the data required by the data consumer is acquired according to the data version which is missing at present of the data consumer, so that when the data consumer and the data producer are in a data transmission interruption period, the data consumer can obtain the data which is possibly evolved by the data producer in the data transmission interruption period in a targeted manner in the normal data transmission process, the missing of the data evolved by the data producer by the data consumer is avoided, and the data transmission quantity and the pressure on the data broadband in the data transmission process are also reduced.
Based on the steps shown in fig. 1, another data obtaining method is proposed in the embodiment of the present invention, in this method, the data version obtaining request may be used to request to obtain all data versions at the data producer, and step S300 may be replaced by steps S330 and S340, as shown in fig. 3:
s330, determining a data version which is next to the currently consumed data version and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a currently effective data version at the data producer as a third data version;
s340, determining the first data version to the third data version as the data version which is currently missing by the data consumer.
The first data version is the next version of the data version currently consumed by the data consumer.
It should be noted that, the present invention directly determines the first data version to the third data version as the data versions currently missing by the data consumer, so that when the data consumer compares the obtained data versions with the data versions already consumed by the data consumer to obtain the currently missing data versions, the data consumer excludes the data versions that are generated earlier than the data version currently consumed by the data consumer and later than the data version that is newly validated at the data producer (these data versions are useless for the data consumer, and the data consumer may not obtain these data versions), thereby further reducing the data transmission amount and the pressure on the data bandwidth during the data transmission process.
According to the data acquisition method provided by the embodiment, the first data version to the third data version are directly determined as the data versions which are currently lost by the data consumer, so that the data corresponding to the data versions which are useless for the data consumer can be further excluded from the data transmission process, and the data transmission quantity and the pressure on the data broadband in the data transmission process are further reduced.
Based on the steps shown in fig. 1, another data obtaining method is proposed in the embodiment of the present invention, as shown in fig. 4, after obtaining the data corresponding to the currently missing data version from the data producer, the method may further include the following steps:
and S500, according to the time sequence of each data version in the current missing data version, sequentially using the data corresponding to the current missing data version to update.
Optionally, in the data obtaining method according to another embodiment of the present invention, as shown in fig. 5, step S500 may include the following steps:
s501, in the current missing data versions, taking the data version which is later than the current consumed data version and is closest to the current consumed data version as the current to-be-updated data version;
s502, updating the data corresponding to the currently consumed data version by using the data corresponding to the currently to-be-updated data version;
s503, determining the current data version to be updated as the current consumed data version, and returning to the step of executing the current missing data version, wherein the data version which is later than the current consumed data version and is closest to the current consumed data version is used as the current data version to be updated.
And the data version which is later than the currently consumed data version and is closest to the currently consumed data version is the next data version of the currently consumed data version of the data consumer.
The invention can sequentially update the data version and the data consumed by the data consumer according to the sequence of the effective time of the data version from front to back. For example: the currently consumed data version of the data consumer is 1.0.1, the marks of the currently missing data versions of the data consumer are 1.0.2,1.0.3, and 1.0.4 respectively (the effective time is 1.0.2,1.0.3, and 1.0.4 from before to after), the data consumer will firstly update according to the data corresponding to the data version 1.0.2, and after the update is completed, the update is respectively performed according to the data in the data versions 1.0.3 and 1.0.4.
When the updating is to be performed or is being performed according to the data version 1.0.2 and the data corresponding to the data version, the data version 1.0.2 is the current data version to be updated; when the update of the data version 1.0.2 and the data corresponding to the data version is completed, the data version 1.0.2 changes the alternative data version 1.0.1 into the data version currently consumed by the data consumer, and the data version 1.0.3 is the data version … … to be updated until the update of the data corresponding to the data version 1.0.4 and the data version 1.0.4 is completed, of course, the data version 1.0.4 also becomes the data version currently consumed by the data consumer after the update is completed.
When a data consumer uses the database to store data, the invention can be carried out in a database transaction mode when carrying out read-write and other operation processes on the database, so that when any process of data processing of the database is wrong, the database can be rolled back to a state before the transaction, the generation of dirty data (useless and repeated data) is avoided, and the situation that a data version updated by the data consumer is not corresponding to corresponding data is also avoided (the update of a certain data version and the update of the data version are the same database transaction).
For example, when an error occurs in the data corresponding to the data version 1.0.4 and the update is not completed, the state (e.g., the amount of the stored data) of the database is rolled back, the data that is just received from the data production place but not completely received is abandoned, and the state of the database is rolled back until the current transaction, that is, the state before the update of the data version 1.0.4 and the data corresponding to the version is returned to the data consumer, the currently consumed data version of the data consumer is 1.0.3, and the currently consumed data of the data consumer is also the data corresponding to the data version 1.0.3.
Or, if an error occurs in the process that the data consumer receives the data corresponding to the data version 1.0.4 sent by the data producer (for example, the data consumer fails to completely receive the data corresponding to the data version due to disconnection of data connection, insufficient local storage space of the data consumer, or other errors), the state of the database of the data consumer is rolled back, the data which is just received from the data producer but not received completely is abandoned, and the state of the database is rolled back to before the transaction, that is, before the data corresponding to the data version 1.0.4 is not received.
According to the data acquisition method provided by the embodiment of the invention, the data consumer can orderly complete the updating of the data version and the corresponding data by sequentially updating the currently missing data versions, so that the data consumer is prevented from missing data when the data version and the corresponding data are updated.
Corresponding to the steps shown in fig. 1, an embodiment of the present invention provides a data acquisition apparatus, which may include an acquisition request transmission unit 100, a first data version acquisition unit 200, a second data version acquisition unit 300, and a data acquisition unit 400, as shown in fig. 6, wherein:
the acquisition request sending unit 100 is configured to send a data version acquisition request to a data producer;
the data consumer is the party receiving and using the data, and the data producer is the party generating and outputting the data.
The data version obtaining request is a request for obtaining the marks of the data versions, and each mark of the data version can uniquely identify one data version. It should be noted that, the present invention is a mark for obtaining the data version when obtaining the data version.
Alternatively, the present invention may use a number to mark different versions of data, or a number representing the time of generation may be used to mark different versions of data.
It should be noted that, when data corresponding to a certain data version (hereinafter referred to as an old data version) evolves, a data producer will create a new data version, where the new data version may be referred to as a next or subsequent data version of the old data version before the evolution of the current data version, and may be referred to as a previous or previous data version of the new data version before the evolution of the current data version.
It should be noted that the data version acquisition request set in the acquisition request sending unit 100 according to the present invention should enable the data consumer to obtain at least the data version after the currently consumed data version. Therefore, the invention can ensure that the data consumer can not always lack the data version and the corresponding data evolved by the data producer.
Optionally, the data corresponding to the new data version may be all data of the data corresponding to the previous data version after the data evolution is completed, or may be only partial data that includes data different from the data corresponding to the previous data version in the all data.
Optionally, the sending a data version obtaining request to a data producer includes:
and sending a data version acquisition request to a data producer according to a preset period.
In practical application, the invention can adopt a Quartz scheduling mode to schedule the work of a data consumer (including sending a data version acquisition request, receiving a data version, receiving data and the like).
The preset period may be set by a technician according to actual conditions, such as an update speed of the data version by the data producer, a demand of the data consumer for a new data version, and the like, which is not limited by the present invention.
The first data version obtaining unit 200 is configured to obtain a data version returned by the data producer according to the data version obtaining request;
it should be noted that, in the acquisition request sending unit 100, the present invention may include the flag of the latest data version currently owned by the data consumer in the information of the data version acquisition request, so that the data producer can determine the data version currently consumed by the data consumer according to the flag in the information, and further send the data version at least after the data version currently consumed by the data consumer to the data consumer.
In practical applications, the present invention may provide at least one interface at the data producer that can receive a data version obtaining request sent by the data consumer.
The method and the system can return the mark of the corresponding data version to the data consumer according to the list of the mark of the data version and/or the format of the object Notation (JSON).
The second data version obtaining unit 300 is configured to compare the obtained data version with a data version already consumed by the data consumer to obtain a data version currently missing by the data consumer;
wherein the data versions currently missing by the data consumer are data versions that the data consumer lags behind the data producer, which may include data versions that have been validated at the data producer and data versions that have not been validated. It should be noted that the data consumer may consume the data corresponding to the validated data version, and cannot consume the data corresponding to the data version that has not been validated yet.
In practice, the present invention can compare the obtained marks of the data versions with the marks of the data versions consumed by the data consumer, respectively, to determine whether the data versions obtained by the present invention are consumed by the data consumer.
Wherein, for a data consumer using the database to store the mark of the data version, the invention can lock the mark of the obtained data version by using a data index mode. And then, with the data version currently consumed by the data consumer as a reference, performing traversal comparison in the obtained data versions, and determining whether each data version is a data version consumed by the data consumer.
The invention can delete the mark which does not belong to the data version which is lost by the data consumer in the current period from the obtained data version when comparing, and only leave the data version which is currently lost by the data consumer.
Alternatively, in the data obtaining apparatus according to another embodiment of the present invention, for the data version obtaining request sent by the data consumer in the obtaining request sending unit 100, the present invention may set it to be used for requesting to obtain all data versions at the data producer, and the second data version obtaining unit 300 may specifically include a first data version determining unit and a second data version determining unit, where:
the first data version determining unit is used for determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version and determining a latest data version at the data producer as a second data version;
the second data version determining unit is used for determining the first data version to the second data version as the data versions which are currently missing by the data consumer.
Thus, the data versions obtained in the first data version determination unit of the present invention are all data versions at the data producer.
The invention determines the data version which is currently missing by the data consumer from all the data versions obtained by the data producer, can ensure that the number of the currently missing data versions obtained by the data consumer is complete, and further ensures that the data consumer can not miss the data version and the data evolved by the data producer.
The data obtaining unit 400 is configured to obtain data corresponding to the currently missing data version from the data producer.
The data producer sends the data obtaining request containing the mark of the current missing data version to the data producer in advance, and the data producer sends the data of the current missing data version corresponding to the data consumer after receiving the data obtaining request.
Optionally, the present invention may be provided with at least one data request and send interface at the data producer, that is, the data consumer may send the data acquisition request to the data producer through the interface, and the data producer may send the data to the data consumer through the interface. The data producer may also set at least one data request interface and at least one data transmission interface, respectively.
It should be noted that, in the implementation process of the present invention, two acquisition requests are sent to the data producer together, where the first acquisition request is about the mark of the data version, and the second acquisition request is about the data version.
The data acquisition device provided by this embodiment sends the data version after the data version currently consumed by the data consumer is acquired to the data producer in advance, determines the currently missing data version of the data consumer according to the acquired data version, and acquires the data required by the data consumer according to the currently missing data version of the data consumer, so that when the data consumer and the data producer are in a data transmission interruption period, the data consumer can obtain the data that the data producer may evolve in the data transmission interruption period in a targeted manner in the subsequent normal data transmission process, the missing of the data evolved by the data producer by the data consumer is avoided, and the data transmission amount and the pressure on the data broadband in the data transmission process are also reduced.
Based on the apparatus shown in fig. 6, another data obtaining apparatus is proposed in the embodiment of the present invention, in which the data version obtaining request may be used to request to obtain all data versions at the data producer, and the second data version obtaining unit 300 may specifically include a third data version determining unit and a fourth data version determining unit, where:
the third data version determining unit is used for determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a currently effective data version at the data producer as a third data version;
the fourth data version determining unit is configured to determine the first to third data versions as data versions currently missing by the data consumer.
The first data version is the next version of the data version currently consumed by the data consumer.
It should be noted that, the present invention directly determines the first data version to the third data version as the data versions currently missing by the data consumer, so that when the data consumer compares the obtained data versions with the data versions already consumed by the data consumer to obtain the currently missing data versions, the data versions with the generation time earlier than the data version currently consumed by the data consumer and later than the data version that is newly effective at the data producer are excluded, thereby further reducing the data transmission amount and the pressure on the data bandwidth during the data transmission process.
The data acquisition device provided by this embodiment directly determines the first data version to the third data version as the data version that is currently missing by the data consumer, and can further exclude data corresponding to the data version that is useless for the data consumer from the data transmission process, thereby further reducing the data transmission amount and the pressure on the data broadband in the data transmission process.
Based on the apparatus shown in fig. 6, another data acquisition apparatus is proposed in the embodiment of the present invention, as shown in fig. 7, the data acquisition apparatus further includes a first data updating unit 500.
The first data updating unit 500 is configured to, after the data corresponding to the currently missing data version is obtained from the data producer, sequentially use the data corresponding to the currently missing data version for updating according to the time sequence of each data version in the currently missing data version.
Optionally, in the data obtaining apparatus according to another embodiment of the present invention, as shown in fig. 8, the first data updating unit 500 may specifically include: a fifth data version determination unit 501, a second data update unit 502, and a return execution unit 503, wherein:
the fifth data version determining unit 501 is configured to, in the currently missing data versions, use a data version that is later than and closest to the currently consumed data version as a current data version to be updated;
the second data updating unit 502 is configured to update data corresponding to a currently consumed data version by using data corresponding to a currently to-be-updated data version;
the return execution unit 503 is configured to determine the current data version to be updated as the current consumed data version, and return to execute the step of taking the data version which is later than the current consumed data version and closest to the current consumed data version as the current data version to be updated in the current missing data versions.
And the data version which is later than the currently consumed data version and is closest to the currently consumed data version is the next data version of the currently consumed data version of the data consumer.
The invention can sequentially update the data version and the data consumed by the data consumer according to the sequence of the effective time of the data version from front to back.
When a data consumer uses the database to store data, the invention can be carried out in a database transaction mode when carrying out read-write and other operation processes on the database, so that when any process of data processing of the database is wrong, the database can be rolled back to a state before transaction, the generation of dirty data is avoided, and the condition that the data version updated by the data consumer does not correspond to the corresponding data is also avoided.
The data acquisition device provided by the embodiment of the invention can enable a data consumer to orderly finish updating the data version and the corresponding data by sequentially updating the currently missing data versions, thereby avoiding the data loss when the data consumer updates the data version and the corresponding data.
Claims (14)
1. A data acquisition method, for application to a data consumer, the method comprising:
sending a data version acquisition request to a data producer;
acquiring a data version returned by the data producer according to the data version acquisition request;
comparing the obtained data version with the consumed data version in the data consumer to obtain the data version which is currently lost by the data consumer;
obtaining data corresponding to the currently missing data version from the data producer.
2. The method of claim 1, wherein sending a data version acquisition request to a data producer comprises:
and sending a data version acquisition request to a data producer according to a preset period.
3. The method of claim 1 or 2, wherein the data version obtaining request is for obtaining all data versions at the data producer.
4. The method of claim 3, wherein comparing the obtained data version with the data versions already consumed in the data consumer to obtain the data version currently missing from the data consumer comprises:
determining a data version subsequent to and adjacent to the currently consumed data version among the obtained data versions as a first data version, and determining a latest data version at the data producer as a second data version;
determining the first data version to the second data version as the data version currently missing by the data consumer.
5. The method of claim 3, wherein comparing the obtained data version with the data versions already consumed in the data consumer to obtain the data version currently missing from the data consumer comprises:
determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a currently effective data version at the data producer as a third data version;
determining the first to third data versions as data versions currently missing by the data consumer.
6. The method of claim 1, further comprising, after said obtaining data corresponding to the currently missing data version from the data producer:
and according to the time sequence of each data version in the current missing data version, sequentially using the data corresponding to the current missing data version to perform updating processing.
7. The method according to claim 6, wherein the sequentially using the data corresponding to the currently missing data version for the update processing according to the time sequence of each data version in the currently missing data version comprises:
taking a data version which is later than the currently consumed data version and is closest to the currently consumed data version as a current data version to be updated in the currently missing data versions;
updating the data corresponding to the currently consumed data version by using the data corresponding to the currently to-be-updated data version;
and determining the current data version to be updated as the current consumed data version, returning to execute the step of taking the data version which is later than the current consumed data version and is closest to the current consumed data version as the current data version to be updated in the current missing data version.
8. A data acquisition apparatus characterized by comprising an acquisition request transmission unit, a first data version acquisition unit, a second data version acquisition unit, and a data acquisition unit, wherein:
the acquisition request sending unit is used for sending a data version acquisition request to a data producer;
the first data version obtaining unit is used for obtaining the data version returned by the data producer according to the data version obtaining request;
the second data version obtaining unit is used for comparing the obtained data version with the consumed data version in the data consumer to obtain the data version which is currently lost by the data consumer;
the data obtaining unit is used for obtaining data corresponding to the current missing data version from the data producer.
9. The data obtaining apparatus according to claim 8, wherein the obtaining request sending unit is specifically configured to send the data version obtaining request to the data producer according to a preset period.
10. The data acquisition apparatus according to claim 8 or 9, wherein the data version acquisition request is for requesting acquisition of all data versions at the data producer.
11. The data acquisition apparatus according to claim 10, wherein the second data version acquisition unit specifically includes a first data version determination unit and a second data version determination unit, wherein:
the first data version determining unit is used for determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version and determining a latest data version at the data producer as a second data version;
the second data version determining unit is used for determining the first data version to the second data version as the data versions which are currently missing by the data consumer.
12. The data obtaining apparatus of claim 10, wherein the second data version obtaining unit specifically includes a third data version determining unit and a fourth data version determining unit, and wherein:
the third data version determining unit is used for determining a data version which is next to and adjacent to the currently consumed data version in the obtained data versions as a first data version, and determining a currently effective data version at the data producer as a third data version;
the fourth data version determining unit is configured to determine the first to third data versions as data versions currently missing by the data consumer.
13. The apparatus according to claim 8, further comprising a first data updating unit, configured to, after the data corresponding to the currently missing data version is obtained from the data producer, sequentially use the data corresponding to the currently missing data version for updating according to a chronological order of the data versions in the currently missing data version.
14. The data acquisition device according to claim 13, wherein the first data update unit specifically includes: a fifth data version determination unit, a second data update unit, and a return execution unit, wherein:
the fifth data version determining unit is used for taking the data version which is later than the currently consumed data version and is closest to the currently consumed data version as the current data version to be updated in the currently missing data versions;
the second data updating unit is used for updating the data corresponding to the currently consumed data version by using the data corresponding to the currently to-be-updated data version;
and the return execution unit is used for determining the current data version to be updated as the current consumed data version, and returning to execute the step of taking the data version which is later than the current consumed data version and is closest to the current consumed data version as the current data version to be updated in the current missing data version.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911002722.XA CN110750547A (en) | 2019-10-21 | 2019-10-21 | Data acquisition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911002722.XA CN110750547A (en) | 2019-10-21 | 2019-10-21 | Data acquisition method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110750547A true CN110750547A (en) | 2020-02-04 |
Family
ID=69279258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911002722.XA Pending CN110750547A (en) | 2019-10-21 | 2019-10-21 | Data acquisition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110750547A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797817A (en) * | 2017-03-13 | 2018-03-13 | 平安科技（深圳）有限公司 | Using update method and device |
| CN108111331A (en) * | 2017-11-06 | 2018-06-01 | 北京趣拿软件科技有限公司 | Acquisition methods, device, storage medium, processor and the system of updated data package |
| CN109542248A (en) * | 2018-11-16 | 2019-03-29 | 上海二三四五网络科技有限公司 | A kind of control method and control device of incremental update dictionary data |
| CN109783135A (en) * | 2019-01-30 | 2019-05-21 | 深圳琥珀虚颜智能科技有限公司 | Multimode software version management system |
| CN110058873A (en) * | 2019-03-12 | 2019-07-26 | 天津五八到家科技有限公司 | Application page update method, device, equipment and storage medium |
-
2019
- 2019-10-21 CN CN201911002722.XA patent/CN110750547A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107797817A (en) * | 2017-03-13 | 2018-03-13 | 平安科技（深圳）有限公司 | Using update method and device |
| CN108111331A (en) * | 2017-11-06 | 2018-06-01 | 北京趣拿软件科技有限公司 | Acquisition methods, device, storage medium, processor and the system of updated data package |
| CN109542248A (en) * | 2018-11-16 | 2019-03-29 | 上海二三四五网络科技有限公司 | A kind of control method and control device of incremental update dictionary data |
| CN109783135A (en) * | 2019-01-30 | 2019-05-21 | 深圳琥珀虚颜智能科技有限公司 | Multimode software version management system |
| CN110058873A (en) * | 2019-03-12 | 2019-07-26 | 天津五八到家科技有限公司 | Application page update method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9245011B2 (en) | Data model versioning for document databases | |
| US20100114818A1 (en) | Method and system for managing and modifying time dependent data structures | |
| JP2015523629A (en) | Mechanisms, systems, and methods for synchronizing devices | |
| CN108647357B (en) | Data query method and device | |
| CN110554732A (en) | identification number generation method and device, electronic equipment and storage medium | |
| US20150067037A1 (en) | Communication apparatus and communication method | |
| CN115599437B (en) | Software version processing method and device, electronic equipment and storage medium | |
| CN106682017B (en) | Database updating method and device | |
| CN110750547A (en) | Data acquisition method and device | |
| CN108762717A (en) | Meter reading success recording method, statistical method and benefit trick method based on Bit-map | |
| CN108491432A (en) | Electric system cumulative amount storage based on eap-message digest and abstracting method, electronic equipment and storage medium | |
| JP2011248852A (en) | System cooperation apparatus | |
| CN111400294A (en) | Data anomaly monitoring method, device and system | |
| JP2017084423A (en) | Information changing apparatus, information changing method, and information changing program | |
| CN107704329B (en) | Message persistence method, server and computer readable storage medium | |
| CN112822098B (en) | Routing method and device of Internet of things and storage medium | |
| CN111090648B (en) | Relational database data synchronization conflict resolution method | |
| CN114169937A (en) | Poster generation method and device, computer equipment and computer readable storage medium | |
| JP2012168870A (en) | Information processing system and form image storage server | |
| CN112446564A (en) | Method, device, equipment and storage medium for determining equipment maintenance time | |
| CN113535430A (en) | Application data read-write separation method and device, computer equipment and storage medium | |
| JP2009295050A (en) | Product management server, product management method and program | |
| JP6546569B2 (en) | Data processing program and data processing method | |
| JP2020181533A (en) | Data collection system and data collection method | |
| CN117370457B (en) | Multithreading data real-time synchronization method, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200204 |
|
| RJ01 | Rejection of invention patent application after publication |