TW201828082A - Data transmission method and device solving the technical problem of frequent failures when full volume of data is updated - Google Patents

Abstract

The present invention discloses a data transmission method and device. The method comprises: receiving periodically, by a first device, a request message from a second device, wherein the request message carries a time stamp corresponding to the request message received this time; determining, by the first device, whether the value of the time stamp is less than a preset threshold value; wherein when the value of the time stamp is less than the preset threshold value, the first device synchronizes sliced data corresponding to the time stamp into the second device according to the corresponding relationship between the time stamp and the sliced data cached locally by the first device, wherein the sliced data is obtained by slicing the full volume of data recorded by the first device. The present invention solves the technical problem of frequent failures when the system updates full volume of data.

Description

 Data transmission method and device  

The present invention relates to the field of Internet technologies, and in particular, to a data transmission method and apparatus.

With the development of network technology, some lawless elements may use the Internet platform to release information content that is harmful to society, such as pornography, gambling, violence, and terrorism, which will adversely affect society. Therefore, it may be necessary to monitor the content of the network, and when such unfavorable information is found, it may be masked to minimize the spread of the unfavorable information.

In the related art, the network content monitoring system generally includes two systems (which can be represented as two devices), for example, system A and system B, and system B needs to synchronize the data in system A, in the related art, each time Synchronization needs to synchronize all data information, which may cause the database to read timeout, or when system A and system B are in different network environments, system A provides too much data, and system B memory is used too much, Causes System B to fail to update data from System A in full.

In response to the above problems, no effective solution has been proposed yet.

Embodiments of the present invention provide a data transmission method and apparatus, so as to at least solve the technical problem of frequent failures when performing full update of data between systems.

According to an aspect of the embodiments of the present invention, a data transmission method is provided, including: a first device periodically receiving a request message from a second device, where the request message carries a timestamp corresponding to the request message received this time The first device determines whether the value of the timestamp is less than a preset threshold; when the value of the timestamp is less than the preset threshold, the correspondence between the first device and the fragmented data of the first device in the local device according to the timestamp, The fragment data corresponding to the time stamp is synchronized to the second device, wherein the fragment data is obtained by fragmenting the full amount of data recorded by the first device.

According to another aspect of the present invention, a data transmission method is further provided, including: the second device sends a request message to the first device, where the request message carries a timestamp corresponding to the request message; The second device receives the fragment data sent by the first device, where the fragment data is the fragment data locally cached by the first device corresponding to the time stamp, and the fragment is the fragmented data. The data is obtained by fragmenting the full amount of data recorded by the first device.

According to another aspect of the present invention, a data transmission apparatus is further provided, which is applied to a first device, and includes: a receiving module, configured to periodically receive a request message from a second device, where the request message carries a timestamp corresponding to the received request message; the first determining module is configured to determine whether the value of the timestamp is less than a preset threshold; the first synchronization module is configured to: the value of the timestamp is less than a preset threshold And synchronizing the fragment data corresponding to the time stamp to the second device according to the correspondence between the time stamp and the fragmented data of the first device in the local cache, wherein the fragment data is the full amount of data recorded by the first device Fragmentation is obtained.

According to another aspect of the present invention, a data transmission apparatus is further provided, which is applied to a second device, and includes: a sending module, configured to send a request message to the first device, where the request message carries a request and a request a time stamp corresponding to the message; the first receiving module is configured to receive the fragment data sent by the first device if the value of the time stamp is less than a preset threshold; wherein the fragment data is corresponding to the time stamp The fragment data of the first device locally cached, and the fragment data is obtained by fragmenting the full amount of data recorded by the first device.

In the embodiment of the present invention, the first device is used to synchronize the locally cached fragment data to the second device, where the fragment data is the full amount recorded by the first device. The method of obtaining the data by sharding, that is, by sharding the full amount of data, and then sequentially synchronizing the shard data in the cache corresponding to the time stamp according to the time stamp in the received request message to the second The way of the device reduces the pressure of data reading, thereby solving the technical problem of frequent failures when the system updates the data in full, and improves the efficiency of data transmission.

10‧‧‧Computer terminal

102‧‧‧Processor

104‧‧‧ memory

106‧‧‧Transmission module

S202, S204, S206‧‧‧ method steps

S302‧‧‧ Method steps

S402, S404, S406, S408, S410, S412, S414, S416, S418, S420, S422, S424, S426‧‧ method steps

S501-S514‧‧‧ method steps

S601, S604‧‧‧ method steps

S702‧‧‧ Method steps

80‧‧‧ receiving module

82‧‧‧First Judgment Module

84‧‧‧First Synchronous Module

90‧‧‧Second judgment module

100‧‧‧Cache Module

1001‧‧‧judging unit

1002‧‧‧First reading unit

1003‧‧‧second reading unit

1004‧‧‧ temporary storage unit

1202‧‧‧Transmission module

1204‧‧‧First Receiver Module

1402‧‧‧second receiving module

1502‧‧‧ processor

1506‧‧‧ memory

1508‧‧‧Transportation device

The drawings are intended to provide a further understanding of the invention, and are intended to be a part of the invention. 1 is a block diagram of a hardware structure of a computer terminal according to a data transmission method according to an embodiment of the present invention; FIG. 2 is a flow chart 1 of a data transmission method according to Embodiment 1 of the present invention; FIG. 4 is a schematic flowchart of a first device fragmentation cache full amount of data according to an alternative embodiment of the present invention; FIG. 5 is a data transmission according to an alternative embodiment of the present invention. FIG. 6 is a flowchart 1 of a data transmission method according to Embodiment 2 of the present invention; FIG. 7 is a flowchart 2 of a data transmission method according to Embodiment 2 of the present invention; FIG. 8 is a flowchart 3 according to Embodiment 1 of the present invention; FIG. 9 is a block diagram of a data transmission device according to Embodiment 3 of the present invention; FIG. 10 is a block diagram of a data transmission device according to Embodiment 3 of the present invention; FIG. 4 is a block diagram of a data transmission device according to Embodiment 3 of the present invention; FIG. 12 is a block diagram of a data transmission device according to Embodiment 4 of the present invention; FIG. 13 is a data according to Embodiment 4 of the present invention; FIG. 14 is a block diagram showing the structure of a data transmission apparatus according to Embodiment 4 of the present invention; and FIG. 15 is a block diagram showing the structure of a computer terminal according to an embodiment of the present invention.

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is an embodiment of the invention, but not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of the present invention.

It should be noted that the terms "first", "second" and the like in the specification and claims of the present invention are used to distinguish similar items, and are not necessarily used to describe a specific order or order. It is to be understood that the materials so used are interchangeable, where appropriate, so that the embodiments of the invention described herein can be carried out in a sequence other than those illustrated or described herein. In addition, the terms "comprises" and "comprises" and "the" and "the" are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to Those steps or units may include other steps or units not explicitly listed or inherent to such processes, methods, products or devices.

In the related art, in the field of network content monitoring and masking, the main steps of data transmission between devices are as follows: 1. When the second device is started, the uniform resource locator url/ is read from the first device for the first time. The function variable name block record; 2. The read uniform resource locator url/function variable name block record is a full update mode, the second device sends a request with a time stamp parameter, the value is 0; After a device receives a request with a timestamp parameter value of 0, it will read out all the url/function variable name blocking records in the current database, and add the current time value (accurate to the minute) and return it to the second. Device; 4. After receiving the full url/function variable name blocking record, the second device clears all blocking records in the system, and uses the full url/function variable name received to block the record; 5. 1 minute later The second device determines whether the current time is a whole point or a half point. If yes, execute 2; if not, send a request to the first device, the time stamped parameter is the time value returned by the first device in 3; 6. the first device receives the request after the timestamp parameter value is not 0 , the data in the database from the time stamp specified time to the current time is read out, and the current time value (accurate to the minute) is added to the second device; 7. The second device receives the incremental url / Function variable name After blocking the record, update all blocked records in the system; then execute 5.

When the url/function variable name blocking data of the first device reaches the tens of millions of levels, the following problem occurs: when the first device reads the url/function variable name blocking data from the database in full, the reading database is caused. Timeout; when the first device and the second device are in different network environments, the network link is unstable; the url/function variable name blocking record of 10 million levels will prolong the data transmission time, often causing the full data transmission failure; The memory of the second device server is limited. When the memory of the second device is used more, the full url/function variable name blocks the exchange of the record, which often causes the memory alarm.

In order to facilitate the understanding of the present invention, the following terms are briefly explained in the embodiments of the present invention: incremental synchronization mode: system A maintains a batch of data that changes instantaneously, system B needs to use the batch of data maintained by system A; when system B When accessing system A to read data, system A only sends the amount of data change between system B's last access and this visit to system B. This data synchronization method is called incremental synchronization mode.

Full-scale synchronization mode: System A maintains a batch of data that changes in real time. System B needs to use the batch of data maintained by System A. When System B accesses System A to read data, System A sends all current data to System B. The data synchronization method is called the full amount synchronization method.

The embodiment of the present invention optimizes the full update process of the blocking records of the first device and the second device, and uses the incremental synchronization mode to analogize the full amount of synchronization to avoid the above problems. The details are described below in conjunction with specific embodiments.

Example 1

In accordance with an embodiment of the present invention, there is also provided an embodiment of a method of data transfer, it being noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer executable instructions, and, although The logical order is shown in the flowchart, but in some cases the steps shown or described may be performed in a different order than here.

The method embodiment provided by Embodiment 1 of the present application can be executed in a mobile terminal, a computer terminal or the like. Taking a computer terminal as an example, FIG. 1 is a hardware block diagram of a computer terminal of a data transmission method according to an embodiment of the present invention. As shown in FIG. 1, computer terminal 10 may include one or more (only one shown) processor 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc. ), a memory 104 for storing data, and a transmission module 106 for communication functions. It will be understood by those skilled in the art that the structure shown in FIG. 1 is merely illustrative and does not limit the structure of the above electronic device. For example, computer terminal 10 may also include more or fewer components than those shown in FIG. 1, or have a different configuration than that shown in FIG.

The memory 104 can be used to store software programs and modules of the application software, such as the program instructions/modules corresponding to the data transmission method in the embodiment of the present invention, and the processor 102 runs the software programs and modules stored in the memory 104. , thereby performing various functional applications and data processing, that is, implementing the data transmission method of the above application. Memory 104 may include high speed random memory and may also include non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, memory 104 can further include memory that is remotely located relative to processor 102, which can be connected to computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission module 106 is configured to receive or transmit data via a network. The above specific network example may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transport module 106 includes a network interface controller (NIC) that can be connected to other network devices through the base station to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module for communicating wirelessly with the Internet.

In the above operating environment, the present application provides a data transmission method as shown in FIG. 2. 2 is a flowchart 1 of a data transmission method according to Embodiment 1 of the present invention. As shown in FIG. 2, the method includes: Step S202: A first device periodically receives a request message from a second device, where the request message is Carrying a timestamp corresponding to the request message received this time; optionally, the first device may be the computer terminal 10 in the above-mentioned operating environment, and the function of the first device is completed through the computer terminal 10 to implement the embodiment. Data transmission method.

It should be noted that the timing of periodically receiving the request message from the second device may be set according to actual conditions, and the time interval of the received request message may be fixed, for example, setting the request message every other minute. It can also be unfixed. For example, after receiving the second request message in one minute, and receiving the third request message after three minutes, the specific request can be set according to the actual situation.

In step S204, the first device determines whether the value of the timestamp is less than a preset threshold. The first device and the second device may be represented by two physical entities, but may be represented by two applications, but are not limited to this. The value of the above time stamp may be a simple serial number such as an Arabic numeral, or may be a time value, but is not limited thereto.

The value of the timestamp may be a specified value for indicating that the full amount of data in the first device is read; in an optional embodiment, the specified value may be 0, that is, the timestamp carried in the request message. A value of 0 indicates that the second device should read the full amount of data of the first device.

In an optional embodiment, before step S204, it is required to determine whether the value of the time stamp is the specified value, and when the determination result is yes, trigger to synchronize the fragment data corresponding to the time stamp to the Second device. In the case that the value of the time stamp is the above specified value, the first device may be triggered to start synchronizing the first fragment data in the full amount of data to the second device.

Step S206: When the value of the timestamp is less than the preset threshold, the first device synchronizes the fragmentation data corresponding to the timestamp to the second according to the correspondence between the timestamp and the fragmented data of the first device in the local cache. The device, wherein the fragmentation data is obtained by fragmenting the full amount of data recorded by the first device.

It should be noted that the preset threshold may be a number of fragments obtained by fragmenting the full amount of data, and may be other values. Generally, the preset threshold may be set to be less than 10; The blocking record of the uniform resource locator url may also be a blocking record for the function variable name, or a combination of the two, and is not limited thereto.

The above fragmentation data can be obtained in the first device by: writing the fragment data corresponding to the first index n in the database of the first device to the local cache, and using n-1 as the fragment data. An index in the local cache of the first device, where n is less than or equal to the total number N, n, and N of all the fragment data obtained after fragmenting the full amount of data. In an optional embodiment, the fragment data corresponding to the first index n in the database of the first device is written into the local cache, and n-1 is used as the local cache of the fragment data in the first device. The index in the specific is: the first device determines whether n is less than N; in the case that n is less than N, the first device reads the fragment data corresponding to n from the database, and attaches the fragment data corresponding to n a timestamp with a value of n; in the case where n is equal to N, the first device reads the fragment data corresponding to n from the database, and attaches the time value of the specified time to the fragment data corresponding to n a stamp; wherein the specified time is the time when the second device accesses the first device last time; the first device writes the fragment data corresponding to n to the local cache, and uses n-1 as the fragment data in the local cache Index in .

It should be noted that the method of performing the cache may be performing the fragment cache according to the transmission format. Therefore, when the second device needs to read, the first device does not need to perform format conversion on the fragment data, but directly returns. For the second device, the synchronization time is reduced and the synchronization efficiency is improved.

By sharding the full amount of data in the first device and then fetching it in the local cache of the first device, each fragment data cached in the cache has a corresponding index, the index and the second device. The time stamp in the sent request message has a certain correspondence relationship, and the fragmented data of the full data fragment is synchronized to the second device by the correspondence between the time stamp and the fragmented data in the cache, in the present invention In an optional embodiment, the synchronizing the fragment data corresponding to the time stamp to the second device in the step S206 may be performed as follows: the first device uses the value of the time stamp as an index, and the local device of the first device is The specified fragment data corresponding to the index is synchronized to the second device.

Through the above steps, by performing the cache sharding of the full amount of data, and according to the time stamp in the received request message, according to the relationship between the time stamp and the shard data in the cache, the time stamp corresponding to the cache is in the cache. The shard data is sequentially synchronized to the second device, which reduces the pressure of data reading, thereby solving the technical problem of frequent failures when updating the data between the systems, and improving the efficiency of data transmission.

For example, a data record of tens of millions of levels is usually recorded in the database of the first device. For the sake of easy understanding, for example, with less data records (80 records), the time stamp of the first device is received as After the request message of 0, according to the related art, the first device synchronizes all 80 records to the second device at one time. In this embodiment, the first device performs fragmentation on the 80 records, and the fragments may be equally distributed or may be distributed unevenly. For example, the equal distribution is divided into four pieces, and each of the fragments has 20 pieces. Record, write the data contained in each slice to the cache, and set the index 0, 1, 2, 3 corresponding to the 4 pieces of data in the cache; when the second device requests to read the full amount of the first device When the data is sent, the time stamp carried in the first request is 0, and the first device synchronizes the fragment data indexed by 0 in the cache to the second device; after a period of time (for example, 1 minute), the first device receives The second request, in which the timestamp is 1 in the second request, the first device synchronizes the fragment data indexed by 1 in the cache to the second device; and so on, until the score in the cache The piece of data has been read. That is, in this embodiment, by sharding the full amount of data, the pieces of data are sequentially synchronized to the second device, that is, the full amount of data is synchronized to the second device in batches, and the reading pressure of the data can be reduced compared with the scheme in the related art. , shorten the transmission time of data, and improve the efficiency of data transmission.

In an alternative embodiment, FIG. 3 is a second flowchart of a data transmission method according to Embodiment 1 of the present invention. As shown in FIG. 3, the method further includes: Step S302, the value of the timestamp is equal to or greater than When the threshold is set, the first device synchronizes the incremental data generated from the time indicated by the time stamp to the current time to the second device.

It should be noted that, in the embodiment of the present invention, there are two types of time stamps, and the first type of time stamp may be the time stamp in the embodiment shown in FIG. 2, and the time stamp may be used to read the scores in the full amount of data. The role of the index of the slice data, which may be represented by a one-digit number such as 0, 1, 2, etc.; the second type of time stamp may be a real time stamp, which represents a real time, that is, the last access of the second device The time of the first device, which can be represented by a 9-bit digit, that is, the second type of time stamp can serve both the identification and the real time. Since the preset threshold is generally fragmented data, it cannot be a large number, so the value of the second type of time stamp that can represent the real time stamp with a 9-bit number is generally greater than the preset threshold (ie, The time stamp equal to or greater than the preset threshold in step S302 can be used as the identifier of the end of the full-quantity data reading or the trigger identifier of the normal incremental data reading.

It should be noted that, in step S302, the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the second device is a normal incremental data read request, and the second device needs to be accessed last time. The incremental data between the time of the first device (ie, the time indicated by the time stamp described above) and the current time is transmitted to the second device, and in the subsequent time, the first device keeps the incremental data in a normal incremental synchronization manner. Synchronizing to the second device until the time point is an hour or a half point, that is, until an indication that the first device synchronizes the data to the second device in a full amount of synchronization is received, and so on.

It should be noted that, in the embodiment of the present invention, the data communication between the first device and the second device may use the http protocol, but is not limited thereto. The above method can be applied to the field of monitoring and masking of network content. For example, the first device monitors network content such as yellow, poison, violence, and terrorism, or other devices monitor the content. After the content is monitored, the uniform resource locator of the webpage corresponding to the content or the function variable name of the computer where the content appears is recorded, and then the data is synchronized to the second device by using the method of the embodiment of the present invention. The second device can be retransmitted to other devices or the second device itself masks the content according to the obtained uniform resource locator or function variable name.

In order to better understand the present invention, the present invention will be explained in one step in conjunction with the optional embodiments.

In an optional embodiment, the first device performs a slice cache on the full url/function variable name in the database of the first device every 30 minutes to solve the read caused by the full reading of the record from the database. Take timeout and other issues. 4 is a schematic flowchart of a first device sharding full data according to an alternative embodiment of the present invention. As shown in FIG. 4, the method includes the following steps: Step S402, setting the full url/function variable name blocking record. The number of fragments (for example, 4), the number of fragments is less than 10; step S404, reading the current time, accurate to the minute, recorded in the timestamp; step S406, reading the total number of records in the current time database according to the value of timestamp (For example, 81); Step S408, calculating the number of records (for example, 20, 20, 20, 21) included in each slice of the full-volume record according to the number of slices; and step S410, setting the slice count index (ie, the above embodiment) The first index n) has a value of 1; in step S412, it is determined whether the current index value is smaller than the number of fragments (corresponding to the total number N of fragmentation materials in the above embodiment); if (for example, 1), the steps are performed. S414; if not (for example, 4), step S422 is performed; step S414, the data included in the specified fragment corresponding to the index is read from the database; step S416, the time stamp is attached, and the value is index; in step S418, the score is Slice data and time stamp In the cache, the key value is index-1 (corresponding to n-1 in the above embodiment); in step S420, the slice count index is incremented by one; the process returns to step S412; and in step S422, the index is read from the database. The specified fragment contains the data; step S424, with a time stamp, the value is timestamp (corresponding to the specified time in the above embodiment); in step S426, the fragment data and the time stamp are written to the cache, and the key value is index. -1.

When the second device reads the full amount of the blocked record from the first device, the first device returns the full amount of the blocked record cache to the second device in batches to solve the network problem that may occur when the large data volume is transmitted once. Problems with the use of the second device memory. 5 is a schematic flowchart of a data transmission method according to an alternative embodiment of the present invention. As shown in FIG. 5, taking a 4-chip cache as an example, the method includes the following steps: Step S501: A second device initiates a request to a first device. Read the full block record, the time stamp is 0; in step S502, the first device determines that the time stamp is 0, is the full update block record, and reads the first slice in the cache of the full block record with 0 as the key. The fragmented data is taken with a timestamp of 1 and returned to the second device; in step S503, the second device clears the blocked record in the system and uses the received full amount of the blocked record in the first cache. Fragment data; step S504, after 1 minute, the second device initiates a request to the first device, and reads the incremental blocking record, the time stamp is recorded as 1 returned by the first device in step S502; in step S505, the first device determines the time. The stamp is less than 10, which is an uncompleted full update block record, and the slice data in the second cache is read in the cache of the full block record with 1 as the key, with time stamp 2, and returned to the second device; Step S506, the second device will receive The fragment data in the second cache is added to the existing blocking record; in step S507, after 1 minute, the second device initiates a request to the first device, and reads the incremental blocking record, and the time stamp is recorded as step S505. The first device returns 2; in step S508, the first device determines that the time stamp is less than 10, which is an uncompleted full update blocking record, and reads the third cache in the cache of the full blocked record with 2 as the key. The fragmentation data is returned to the second device with the time stamp 3; in step S509, the second device adds the fragmentation data in the received third cache to the existing blocking record; step S510, 1 After the minute, the second device initiates a request to the first device to read the incremental blocking record, and the timestamp is recorded as 3 returned by the first device in step S508; in step S511, the first device determines that the timestamp is less than 10, which is an unfinished full amount. Update the blocking record, read the fragment data in the fourth cache in the cache of the full blocking record with 3 as the key, and attach the timestamp to the timestamp of the full blocking recording cache, and return to the second Device; step S512, the second device will The received fragment data in the fourth cache is added to the existing blocking record; in step S513, after 1 minute, the second device initiates a request to the Mrm, reads the incremental blocking record, and the time stamp is step The timestamp returned by the first device in S511; in step S514, the first device Mrm determines that the timestamp is greater than 10, which is a normal incremental blocking record request, and reads the change of the blocked record from the timestamp in the database to the current time, with the current The time is returned to the second device; the second device performs an incremental blocking record request every minute; if it is an hour or a half point, the second device does not perform the incremental blocking record request, and returns to step S501.

It should be noted that 1 minute in the optional embodiment shown in FIG. 5 of the present invention corresponds to the timing of the embodiment shown in FIG. 2 of the above embodiment, and the time stamp is 0, which is equivalent to the embodiment in the embodiment shown in FIG. 2 . The value of the timestamp is a specified value to indicate that the full-time blocking record is read. The timestamp in the above embodiment is 0, 1, 2, and 3, which is equivalent to the value of the timestamp in the embodiment shown in FIG. 2; Corresponding to the index in the embodiment shown in FIG. 2, the timestamp is equivalent to the timestamp in which the value in the embodiment shown in FIG. 3 is equal to or greater than a preset threshold.

In the above embodiment, the synchronization is divided into two types, an incremental synchronization mode and a full-quantity synchronization mode. The full-quantity synchronization mode synchronization data is realized by: pre-sending the full amount of data in advance to perform fragmentation cache, and reading in the second device. At the time, the fragmentation data in the fragment cache is directly used, that is, according to the number N of the full-size data fragments, the full-scale data synchronization is realized by the N-class analog incremental synchronization method, thereby reducing the data reading pressure in the first device. When the second device is synchronized to the second device, the synchronization will not fail because the data volume is too large.

It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present invention is not limited by the described action sequence. Because certain steps may be performed in other sequences or concurrently in accordance with the present invention. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

Through the description of the above embodiments, those skilled in the art can clearly understand that the data transmission method according to the above embodiment can be implemented by means of a software plus a necessary general hardware platform, and of course, can also be through hardware, but many In the case of the former is a better implementation. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, can be embodied in the form of a software product stored in a storage medium (such as ROM/RAM, disk, CD-ROM). The instructions include a number of instructions for causing a terminal device (which may be a cell phone, a computer, a server, or a network device, etc.) to perform the methods described in various embodiments of the present invention.

Example 2

According to an embodiment of the present invention, a data transmission method is also provided. FIG. 6 is a flowchart 1 of a data transmission method according to Embodiment 2 of the present invention. As shown in FIG. 6, the method includes: Step S602, the second device is directed to A device sends a request message, wherein the request message carries a timestamp corresponding to the request message. It should be noted that the second device may be a computer terminal having the same structure as the computer terminal 10 in the first embodiment. The data transmission method in this embodiment is completed in the environment of the computer terminal, but is not limited thereto. The foregoing sending request message may be a timing sending request message, and may be a request message sent from time to time, and may be specifically set according to a specific situation.

Step S604: The second device receives the fragment data sent by the first device, where the value of the timestamp is less than a preset threshold, where the fragment data is locally cached in the first device corresponding to the timestamp. The fragmentation data is obtained by fragmenting the full amount of data recorded by the first device.

The value of the timestamp may be a specified value for instructing the second device to read the full amount of data in the first device. In an optional embodiment, the specified value may be 0, that is, carried in the request message. When the timestamp is 0, it indicates that the second device should read the full amount of data of the first device. In an optional embodiment, before the step S604, the method further includes: when the time stamp is a specified value, the second device triggers the first device to send the full amount of data, and the first device sends the corresponding value corresponding to the specified value. Segmentation information.

It should be noted that the preset threshold may be the number of fragments obtained by the first device after fragmenting the full amount of data, and may of course be other values. Generally, the preset threshold may be set to be less than 10; It may be a blocking record for the uniform resource locator url, a blocking record for the function variable name, or a combination of the two, and is not limited thereto.

Specifically, the first device performs the fragmentation cache on the full amount of data, that is, how the fragment data is obtained in the first device, which is the same as that in the first embodiment, and details are not described herein again.

In the foregoing step S604, the receiving, by the second device, the fragment data sent by the first device may be: the second device receives the specified fragment data corresponding to the index in the local cache of the first device; wherein the index is the time stamp The second device receives the specified timestamp sent by the first device; wherein the value of the specified timestamp is obtained by adding 1 to the value of the timestamp.

Through the foregoing steps, the request message carrying the time stamp is sent by the second device, and the manner in which the first device sends the fragment data in the cache of the first device corresponding to the time stamp according to the time stamp in the received request message is received. That is, the second device reads the fragment data that is pre-sorted and cached in the first device according to the time stamp, thereby reducing the pressure of data reading, thereby solving the technical problem of frequent failures when the system updates the data in full. Improve the efficiency of data transmission.

For example, a data record of tens of millions of levels is usually recorded in the database of the first device. For the sake of easy understanding, for example, with less data records (80 records), the second device sends a time stamp of 0. After requesting the message, the second device will read the 80 records at a time according to the manner in the related art. In this embodiment, the first device performs fragmentation on the 80 records, and the fragments may be equally distributed or may be distributed unevenly. For example, the equal distribution is divided into four pieces, and each of the fragments has 20 pieces. Record, write the data contained in each slice to the cache, and set the index 0, 1, 2, 3 corresponding to the 4 pieces of data in the cache; when the second device requests to read the full amount of the first device When the data is sent, the time stamp carried in the first request is 0, and the second device reads the fragment data indexed by 0 in the cache of the first device; after a period of time (for example, 1 minute), the second device sends The second request, the second request carries a timestamp of 1, and the second device reads the fragment data indexed by 1 in the cache of the first device; and so on, until the second device will be the first The fragment data in the cache of the device is read. That is, in this embodiment, the full amount of data is fragmented by the first device, and the second device reads the fragmentation data in the cache of the first device in batches, which can reduce the reading pressure of the data compared to the scheme in the related art. , shorten the transmission time of data, and improve the efficiency of data transmission.

In an alternative embodiment, FIG. 7 is a flowchart 2 of a data transmission method according to Embodiment 2 of the present invention. As shown in FIG. 7, the method further includes: Step S702, the value of the timestamp is equal to or greater than In the case of setting the threshold, the second device receives the incremental data generated by the first device from the time indicated by the time stamp to the current time.

It should be noted that, in the embodiment of the present invention, there are two types of time stamps, and the first type of time stamp may be the time stamp in the embodiment shown in FIG. 6 , and the time stamp may be used to read the scores in the full amount of data. The role of the index of the slice data, which may be represented by a one-digit number such as 0, 1, 2, etc.; the second type of time stamp may be a real time stamp, which represents a real time, that is, the last access of the second device The time of the first device, which can be represented by a 9-bit digit, that is, the second type of time stamp can serve both the identification and the real time. Since the preset threshold is generally fragmented data, it is not a large number of digits, so the second type of time stamp, which can represent the real time stamp with 9 digits, is generally greater than the preset threshold (ie, in step S702). The time stamp equal to or greater than the preset threshold, so that the second type of time stamp can be used as an identifier for the end of a full amount of data reading, or a trigger identifier for a normal incremental data reading.

It should be noted that, in step S702, the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the second device is a normal incremental data read request, and the second device needs to use the second device. The incremental data between the time when the last access to the first device (that is, the time indicated by the time stamp described above) is up to the current time is read, and in the subsequent time, the second device is always read in the normal incremental synchronization manner. Incremental data until the point in time is the hour or half, that is, until the second device receives an indication to read the data in full synchronous mode, and so on.

It should be noted that, in the embodiment of the present invention, the data communication between the first device and the second device may use the http protocol, but is not limited thereto. The above method can be applied to the field of monitoring and masking of network content. For example, the first device monitors content of the network such as yellow, poison, violence, and terrorism, or other devices perform the content. Monitoring, when the content is monitored, the uniform resource locator of the webpage corresponding to the content or the function variable name of the computer where the content appears is recorded, and then the data is synchronized to the second device by using the method of the embodiment of the present invention. The second device can be retransmitted to other devices or the second device itself masks the content according to the obtained uniform resource locator or function variable name.

For a better understanding of the present invention, the present invention can be explained in one step in combination with the optional embodiments shown in FIG. 4 and FIG. 5 in Embodiment 1, and details are not described herein again.

Example 3

An apparatus for implementing the data transmission method in Embodiment 1 is provided according to the embodiment of the present invention. FIG. 8 is a block diagram of the structure of the data transmission apparatus according to Embodiment 3 of the present invention, as shown in FIG. The device is applied to the first device, and includes: a receiving module 80, configured to periodically receive a request message from the second device, where the request message carries a timestamp corresponding to the request message received this time; The timing here can be set according to the actual situation. The time interval of the request message received twice can be fixed. For example, the request message is received every 1 minute, or it is not fixed, for example, after 1 minute. After receiving the second request message, the third request message is received after 3 minutes, and the specific request can be set according to the actual situation.

The first judging module 82 is connected to the receiving module 80, and is configured to determine whether the value of the timestamp is less than a preset threshold; the value of the timestamp may be a specified value for indicating reading in the first device. The full amount of data; in an optional embodiment, the specified value may be 0, that is, when the timestamp carried in the request message is 0, the second device is instructed to read the full amount of data of the first device.

In an alternative embodiment, FIG. 9 is a block diagram of a data transmission device according to Embodiment 3 of the present invention. As shown in FIG. 9, the device further includes: a second determining module 90, and a receiving module 80. a connection, configured to determine whether the value of the timestamp is a specified value, wherein the specified value is used to indicate that the full amount of data is read, wherein, when the determination result is yes, triggering to synchronize the fragment data corresponding to the timestamp to the first Two devices.

The first synchronization module 84 is connected to the first judging module 82, and is configured to, according to the timestamp, the correspondence between the time stamp and the fragmented data of the first device in the local cache when the value of the timestamp is less than the preset threshold. The fragment data corresponding to the time stamp is synchronized to the second device, wherein the fragment data is obtained by fragmenting the full amount of data recorded by the first device.

It should be noted that the preset threshold may be a number of fragments obtained by fragmenting the full amount of data, and may be other values. Generally, the preset threshold may be set to be less than 10; The blocking record of the uniform resource locator url may also be a blocking record for the function variable name, or a combination of the two, and is not limited thereto.

The first synchronization module 84 is further configured to synchronize the specified fragment data corresponding to the index in the local cache of the first device to the second device by using the value of the time stamp as an index.

In an alternative embodiment, FIG. 10 is a structural block diagram of a data transmission apparatus according to Embodiment 3 of the present invention. As shown in FIG. 10, the apparatus further includes: a cache module 100, and a first synchronization mode. The group 84 is connected to write the fragment data corresponding to the first index n in the database of the first device to the local cache, and use n-1 as the fragment data in the local cache of the first device. The index, where n is less than or equal to the total number N, n, and N of all the pieces of data obtained after fragmenting the full amount of data are positive integers.

Specifically, the cache module 100 further includes: a determining unit 1001, configured to determine whether n is less than N; the first reading unit 1002 is connected to the determining unit 1001, and is configured to use n in the case where n is smaller than N. The fragment data corresponding to n is read in the database, and a time stamp with a value of n is attached to the fragment data corresponding to n; the second reading unit 1003 is connected to the determining unit 1001 for equalizing In the case of N, the fragment data corresponding to n is read from the database, and a time stamp having a value of the specified time is attached to the fragment data corresponding to n; wherein the specified time is the The time when the second device accesses the first device at a time; the temporary storage unit 1004 is connected to the first reading unit 1002 and the second reading unit 1003, and is configured to write the fragment data corresponding to n to the local fast. Take and use n-1 as the index of the fragmented data in the local cache.

It should be noted that the temporary storage unit 1004 may write the fragment data corresponding to n to the local cache according to the transmission format, so that the first device does not need to re-segment the data when the second device needs to read. The format is changed, but directly returned to the second device, which reduces the synchronization time and improves the synchronization efficiency.

Through the above device, the cache is correspondingly cached according to the time stamp of the received request message, according to the time stamp in the received request message, and the relationship between the time stamp and the snapshot data in the cache. The shard data is sequentially synchronized to the second device, which reduces the pressure of data reading, thereby solving the technical problem of frequent failures when updating the data between the systems, and improving the efficiency of data transmission.

For example, a data record of tens of millions of levels is usually recorded in the database of the first device. For the sake of easy understanding, for example, with less data records (80 records), the time stamp of the first device is received as After the request message of 0, according to the related art, the first device synchronizes all 80 records to the second device at one time. In the embodiment, the device in the first device fragments the 80 records, and the fragments can be equally distributed or distributed unevenly. For example, the equal distribution is divided into four pieces, each of which is divided into four pieces. The slice has 20 records, and the temporary storage unit 1004 writes the data contained in each slice to the cache, and sets the index 0, 1, 2, 3 corresponding to the 4 pieces of data in the cache; when the second device When requesting to read the full amount of data of the first device, the time stamp carried in the first request is 0, and the first synchronization module 84 of the first device synchronizes the fragment data indexed by 0 in the cache to the second After a period of time (for example, 1 minute), the receiving module 80 of the first device receives the second request, and the time stamp of the second request is 1; then the first synchronization module 84 of the first device will be fast. The fragment data indexed by 1 is synchronized to the second device; and so on, until the fragment data in the cache is read. That is, the device of the embodiment can perform the fragmentation of the full amount of data to the second device in sequence, that is, the full amount of data is synchronously synchronized to the second device, and the data can be reduced compared to the solution in the related art. The reading pressure shortens the data transmission time and improves the data transmission efficiency.

Figure 11 is a block diagram showing the structure of a data transmission device according to a third embodiment of the present invention. As shown in Figure 11, the device further includes a second synchronization module 1101 connected to the first determination module 82 for When the value of the time stamp is equal to or greater than the preset threshold, the incremental data generated from the time indicated by the time stamp to the current time is synchronized to the second device.

It should be noted that, in the embodiment of the present invention, there are two types of time stamps, and the first type of time stamp may be the time stamp in the embodiment shown in FIG. 8 , and the time stamp can be used to read the scores in the full amount of data. The role of the index of the slice data, which may be represented by a one-digit number such as 0, 1, 2, etc.; the second type of time stamp may be a real time stamp, which represents a real time, that is, the last access of the second device The time of the first device, which can be represented by a 9-bit digit, that is, the second type of time stamp can serve both the identification and the real time. Since the preset threshold is generally fragmented data, it may not be a large number of digits, and thus the value of the second type of time stamp, which can represent the real time stamp with 9 digits, is generally greater than the preset threshold. Therefore, the second type time stamp can be used as an identifier for the end of a full data read, or a trigger identifier for a normal incremental data read.

It should be noted that the value of the timestamp is greater than or equal to the preset threshold, indicating that the request sent by the second device is a normal incremental data read request. At this time, the second synchronization module 1101 needs to bring the second device closest. The incremental data between the time when the first device is accessed (ie, the time indicated by the time stamp described above) to the current time is transmitted to the second device, and in the subsequent time, the first device always uses the normal incremental data synchronization mode. The incremental data is synchronized to the second device until the time point is an hour or a half point, that is, until an indication that the first device synchronizes the data to the second device in a full amount of read sync mode is received, and so on.

It should be noted that, in the embodiment of the present invention, the data communication between the foregoing device and the second device in the first device may use the http protocol, but is not limited thereto. The above device can be applied to the field of monitoring and masking of network content. For example, the first device monitors content of the network such as yellow, poison, violence, and fearful content, or other devices perform the content. Monitoring, after monitoring the content, recording the uniform resource locator of the webpage corresponding to the content or the function variable name of the computer where the content appears, and then synchronizing the data to the second device through the device, the second device It can be retransmitted to other devices or the second device itself masks the content according to the obtained Uniform Resource Locator or Function Variable Name.

It should be noted that the second synchronization module 1101 of the first synchronization module 84 may be the same synchronization module in the first device, or may be a different synchronization module in the first device.

Example 4

An apparatus for implementing the data transmission method in Embodiment 2 is further provided. FIG. 12 is a block diagram of the structure of the data transmission apparatus according to Embodiment 4 of the present invention. As shown in FIG. The device is applied to the second device, and includes: a sending module 1202, configured to send a request message to the first device, where the request message carries a timestamp corresponding to the request message; The message may be a timed request message, or may be sent from time to time, and may be specifically set according to specific conditions.

The first receiving module 1204 is connected to the sending module 1202, and configured to receive the fragment data sent by the first device if the value of the time stamp is less than a preset threshold; wherein the fragment data is a time stamp Corresponding the fragmented data locally cached by the first device, the fragmented data is obtained by fragmenting the full amount of data recorded by the first device.

The timestamp may be a specified value for instructing the second device to read the full amount of data in the first device; in an optional embodiment, the specified value may be 0, that is, the timestamp carried in the request message is 0, indicating that the second device should read the full amount of data of the first device. In an optional embodiment, the device further includes: a triggering module, configured to: when the value of the timestamp is a specified value, trigger the first device to send the full amount of data; the third receiving module, and the triggering mode The group connection is configured to receive the fragment data corresponding to the specified value sent by the first device, where the specified value is used to instruct the second device to read the full amount of data to the first device.

It should be noted that the preset threshold may be the number of fragments obtained by the first device after fragmenting the full amount of data, and may of course be other values. Generally, the preset threshold may be set to be less than 10; It may be a blocking record for the uniform resource locator url, a blocking record for the function variable name, or a combination of the two, and is not limited thereto.

Specifically, the first device performs the fragmentation cache on the full amount of data, that is, how the fragment data is obtained in the first device, which is the same as that in the first embodiment, and details are not described herein again.

In an alternative embodiment of the present invention, FIG. 13 is a block diagram of a data transmission apparatus according to Embodiment 4 of the present invention. As shown in FIG. 13, the first receiving module 1204 may include: a first receiving unit. The first receiving unit 1304 is configured to receive the specified fragment data corresponding to the index in the local cache of the first device, where the index is the value of the time stamp; the second receiving unit 1304 is connected to the first receiving unit 1302. And receiving the specified timestamp sent by the first device; wherein the value of the specified timestamp is obtained by adding 1 to the value of the timestamp.

The sending module 1202 sends a request message carrying a time stamp through the sending module 1202, and the first receiving module 1204 receives the first device's cache corresponding to the timestamp corresponding to the timestamp sent by the first device according to the received request message. The method of sharding data in the device, that is, the device reads the shard data of the sharding and sharding in the first device according to the time stamp, thereby reducing the pressure of reading the data, thereby solving the problem of fully updating the data between the systems. Technical problems that frequently fail, improve the efficiency of data transmission.

For example, a data record of tens of millions of levels is usually recorded in the database of the first device. For the sake of easy understanding, for example, with less data records (80 records), the second device sends a time stamp of 0. After requesting the message, the second device will read the 80 records at a time according to the manner in the related art. In this embodiment, the first device performs fragmentation on the 80 records, and the fragments may be equally distributed or may be distributed unevenly. For example, the equal distribution is divided into four pieces, and each of the fragments has 20 pieces. Record, write the data contained in each slice to the cache, and set the index 0, 1, 2, 3 corresponding to the 4 pieces of data in the cache; when the second device is in the above device When the 1202 requests to read the full amount of data of the first device, the timestamp carried in the first request is 0, and the first receiving module 1204 in the device reads the index indexed by 0 in the cache of the first device. After a period of time (for example, 1 minute), the sending module 1202 in the device of the second device sends a second request, and the second request carries a time stamp of 1, and the first receiving mode in the device The group 1204 reads the fragment data indexed by 1 in the cache of the first device; and so on, until the device in the second device reads the fragment data in the cache of the first device. That is, in this embodiment, the full amount of data is fragmented by the first device, and the device reads the fragment data in the cache of the first device in batches, which can reduce the reading pressure of the data compared to the scheme in the related art. Shorten the transmission time of data and improve the efficiency of data transmission.

FIG. 14 is a block diagram showing the structure of a data transmission apparatus according to Embodiment 4 of the present invention. As shown in FIG. 14, the apparatus further includes: a second receiving module 1402, and a value used by the sending module 1204 for time stamping. When the threshold is greater than or equal to the preset threshold, the incremental data generated by the first device from the time indicated by the time stamp to the current time is received.

It should be noted that, in the embodiment of the present invention, there are two types of time stamps, and the first type of time stamp may be the time stamp in the embodiment shown in FIG. 12, and the time stamp may be used to read the scores in the full amount of data. The role of the index of the slice data, which may be represented by a one-digit number such as 0, 1, 2, etc.; the second type of time stamp may be a real time stamp, which represents a real time, that is, the last access of the second device The time of the first device, which can be represented by a 9-bit digit, that is, the second type of time stamp can serve both the identification and the real time. Since the preset threshold is generally fragmented data, it is not a large number of digits, so the second type of time stamp, which can represent the real time stamp with 9 digits, is generally larger than the preset threshold, and thus can be The second type of time stamp is used as an identifier for the end of a full data read, or an indication of the beginning of a normal incremental data read.

It should be noted that the value of the timestamp is greater than or equal to the preset threshold, and the request sent by the device in the second device is a normal incremental data read request. At this time, the device of the second device needs to be The incremental data between the time when the second device last accessed the first device (ie, the time indicated by the time stamp described above) to the current time is read, and in the subsequent time, the device is always synchronized with the normal incremental data. The method reads the incremental data until the time point is the whole point or half point, that is, until the above device receives the instruction to read the data in full reading synchronous mode, and so on.

It should be noted that, in the embodiment of the present invention, the data communication between the first device and the foregoing device in the second device may use the http protocol, but is not limited thereto. The above device can be applied to the field of monitoring and masking of network content. For example, the first device monitors content of the network such as yellow, poison, violence, and fearful content, or other devices perform the content. Monitoring, after monitoring the content, recording the uniform resource locator of the webpage corresponding to the content or the function variable name of the computer where the content appears, and then synchronizing the data to the second device through the device, the second device It can be retransmitted to other devices or the second device itself masks the content according to the obtained Uniform Resource Locator or Function Variable Name.

Example 5

An embodiment of the present invention may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.

In this embodiment, the computer terminal can execute the code of the step in the data transmission method in the first embodiment of the application. For details, refer to the embodiment 1, and details are not described herein again.

Optionally, FIG. 15 is a block diagram showing the structure of a computer terminal according to an embodiment of the present invention. As shown in FIG. 15, the computer terminal A may include one or more (only one shown in the figure) processor 1502, memory 1506, and transmission device 1508.

The memory 1506 can be used to store software programs and modules, such as the security vulnerability detection method and the program instruction/module corresponding to the device in the embodiment of the present invention. The processor 1502 runs the software program and the module stored in the memory. The group, thereby performing various functional applications and data processing, that is, the detection method for implementing the above system vulnerability attack. The memory 1506 can include high speed random memory and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid state memory. In some examples, the memory can further include a memory 1506 that is remotely located relative to the processor 1502, and the remote registers 1506 can be connected to the terminal A via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor 1502 can invoke the information and application stored in the memory 1506 through the transmission device 1408 to execute the application in the steps in Embodiment 1.

With the embodiment of the invention, a solution of a computer terminal is provided. Through the computer terminal, the code of the steps in the first embodiment is executed, the pressure of data reading is reduced, and the technical problem of frequent failures when updating the data between the systems is solved, and the efficiency of data transmission is improved.

A person skilled in the art can understand that the structure shown in FIG. 15 is only an illustration, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (MID). ), PAD and other terminal devices. Fig. 15 does not limit the structure of the above electronic device. For example, computer terminal A may also include more or fewer components (such as a network interface, display device, etc.) than shown in FIG. 15, or have a different configuration than that shown in FIG.

A person skilled in the art can understand that all or part of the steps of the foregoing embodiments can be completed by using a program to instruct a terminal device related hardware, and the program can be stored in a computer readable storage medium, and the storage medium can be Including: flash memory disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk.

Example 6

An embodiment of the present invention may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the foregoing computer terminal may also be replaced with a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device of the plurality of network devices of the computer network.

In this embodiment, the computer terminal can execute the code of the step in the data transmission method in the second embodiment of the application. For details, see Embodiment 2, and details are not described herein again.

The computer terminal in this embodiment is similar in structure to the computer terminal in the embodiment 5, and includes a processor, a memory, and a transmission device, wherein the memory is used to store the code of the step of the data transmission method in the second embodiment. The processor is configured to invoke the information and application stored in the memory through the transmission device to execute the code of the steps in Embodiment 2. For details, refer to the method in Embodiment 2, and details are not described herein again.

Example 7

Embodiments of the present invention also provide a storage medium. Optionally, in the embodiment, the storage medium may be used to save the code executed by the data transmission method provided in the first embodiment.

Optionally, in this embodiment, the storage medium may be located in any one of the computer terminal groups in the computer network, or in any one of the mobile terminal groups.

Optionally, in this embodiment, the storage medium is configured to store a code for performing the steps of the data transmission method in Embodiment 1 or Embodiment 2. For details, see Embodiment 1 or Embodiment 2, I won't go into details here.

The serial numbers of the embodiments of the present invention are merely for the description, and do not represent the advantages and disadvantages of the embodiments.

In the above-mentioned embodiments of the present invention, the descriptions of the various embodiments are different, and the parts that are not detailed in a certain embodiment can be referred to the related descriptions of other embodiments.

In the several embodiments provided by the present application, it should be understood that the disclosed technical contents may be implemented in other manners. The device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or integrated. Go to another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, unit or module, and may be electrical or otherwise.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of a hardware or a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, can be stored in a computer readable storage medium. Based on such an understanding, the technical solution of the present invention may contribute to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: U disk, read-only memory (ROM), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disc, etc. Media.

The above is only a preferred embodiment of the present invention, and it should be noted that those skilled in the art can make several improvements and retouchings without departing from the principles of the present invention. It should also be considered as the scope of protection of the present invention.

Claims (24)

 A data transmission method, the method includes: the first device periodically receives a request message from the second device, where the request message carries a time stamp corresponding to the request message received this time; The device determines whether the value of the timestamp is less than a preset threshold; and when the value of the timestamp is less than the preset threshold, the first device fetches the fragmented data locally with the first device according to the timestamp. Corresponding relationship, the fragment data corresponding to the time stamp is synchronized to the second device, wherein the fragment data is obtained by fragmenting the full amount of data recorded by the first device.    The method of claim 1, wherein the method further comprises: when the value of the time stamp is equal to or greater than the preset threshold, the first device will use the time indicated by the time stamp to the current time. The generated incremental data is synchronized to the second device.    The method of claim 1, wherein the method further comprises: determining, by the first device, whether the value of the timestamp is the value of the timestamp, if the first device determines whether the value of the timestamp is less than a preset threshold. And a specified value, wherein the specified value is used to indicate that the full amount of data is read; and, when the determination result is yes, triggering to synchronize the fragment data corresponding to the time stamp to the second device.    The method of claim 1, wherein the synchronizing the fragment data corresponding to the time stamp to the second device comprises: the first device using the value of the time stamp as an index, The specified fragment data corresponding to the index in the local cache of a device is synchronized to the second device.    The method of claim 1, wherein the fragmentation data is obtained in the first device by the first device: the first device corresponding to the first index n in the database of the first device The slice data is written to the local cache, and n-1 is used as an index of the fragment data in the local cache of the first device, where n is less than or equal to all obtained after fragmenting the full amount of data. The total number of pieces of data N, n, and N are positive integers.    The method of claim 5, wherein the first device writes the fragment data corresponding to the first index n in the database of the first device to the local cache, and n- The index of the fragment data in the local cache of the first device includes: the first device determines whether n is less than N; and in the case where n is less than N, the first device reads n from the database. Corresponding fragment data, and a time stamp with a value of n is attached to the fragment data corresponding to n; in the case where n is equal to N, the first device reads the fragment data corresponding to n from the database. And attaching, to the fragment data corresponding to n, a time stamp whose value is a specified time; wherein the specified time is a time when the second device accesses the first device last time; and the first device corresponds to n The fragment data is written to the local cache, and n-1 is used as an index of the fragment data in the local cache.    The method of any one of claims 1 to 6, wherein the full amount of data comprises at least one of: a blocking record for a uniform resource locator, a blocking record for a function variable name.    A data transmission method, the method includes: the second device sends a request message to the first device, where the request message carries a timestamp corresponding to the request message; and the value of the timestamp is less than In the case of a preset threshold, the second device receives the fragment data sent by the first device, where the fragment data is a fragment data locally cached by the first device corresponding to the time stamp. The piece of data is obtained by fragmenting the full amount of data recorded by the first device.    The method of claim 8, wherein the method further comprises: if the value of the time stamp is equal to or greater than the preset threshold, the second device receives the The time indicated by the timestamp to the incremental data generated by the current time.    The method of claim 8, wherein the method further comprises: before the second device receives the fragmented data of the first device that is locally cached by the first device, the method further includes: If the value of the time stamp is a specified value, the second device triggers the first device to send the full amount of data, and receives the fragment data corresponding to the specified value sent by the first device, where the designation The value is used to instruct the second device to read the full amount of data to the first device.    The method of claim 8, wherein the receiving, by the second device, the fragmentation data sent by the first device comprises: the second device receiving the designation corresponding to the index in the local cache of the first device a fragmentation data; wherein the index is a value of the timestamp; and the second device receives the specified timestamp sent by the first device; wherein the value of the specified timestamp is determined by the value of the timestamp 1 get.    The method of any one of clauses 8 to 11, wherein the full amount of data comprises at least one of: a blocking record for a uniform resource locator, a blocking record for a function variable name.    A data transmission device is applied to a first device, and the device includes: a receiving module, configured to periodically receive a request message from a second device, where the request message carries the request received this time a timestamp corresponding to the message; the first determining module is configured to determine whether the value of the timestamp is less than a preset threshold; and the first synchronization module is configured to: when the value of the timestamp is less than the preset threshold, And synchronizing the fragment data corresponding to the time stamp to the second device according to the correspondence between the time stamp and the fragmented data of the first device in the local cache, wherein the fragment data is the first The full amount of data recorded by the device is obtained by fragmentation.    The device of claim 13, wherein the device further comprises: a second synchronization module, when the time stamp is equal to or greater than the preset threshold, from the time indicated by the time stamp to the current time The incremental data generated by the time is synchronized to the second device.    The device of claim 13, wherein the device further comprises: a second determining module, configured to determine whether the value of the time stamp is a specified value, wherein the specified value is used to indicate that the reading is performed The full amount of data, wherein, when the determination result is yes, triggering to synchronize the fragment data corresponding to the time stamp to the second device.    The device according to claim 13 , wherein the first synchronization module is further configured to use the value of the time stamp as an index to specify the local cache of the first device corresponding to the index. The fragmentation data is synchronized to the second device.    The device of claim 13, wherein the device further comprises: a cache module, configured to write the fragment data corresponding to the first index n in the database of the first device to the local device Cache, and use n-1 as an index of the fragment data in the local cache of the first device, where n is less than or equal to the total number of all the fragment data obtained after fragmenting the full amount of data. , n, N are positive integers.    The device of claim 17, wherein the cache module comprises: a judging unit configured to determine whether n is less than N; and a first reading unit configured to: when n is less than N, The data corresponding to n is read in the database, and a time stamp with a value of n is attached to the fragment data corresponding to n; the second reading unit is configured to use the database when n is equal to N. Reading the fragment data corresponding to n, and attaching a timestamp value of the specified time to the fragment data corresponding to n; wherein the specified time is the time when the second device accesses the first device last time And a temporary storage unit, configured to write the fragment data corresponding to n to the local cache, and use n-1 as an index of the fragment data in the local cache.    The apparatus of any one of claims 13 to 18, wherein the full amount of data comprises at least one of: a blocked record for a uniform resource locator, a blocked record for a function variable name.    A data transmission device is applied to a second device, and the device includes: a sending module, configured to send a request message to the first device, where the request message carries a time stamp corresponding to the request message; And receiving, by the first receiving module, the fragmentation data sent by the first device, where the value of the timestamp is less than a preset threshold; wherein the fragmentation data is corresponding to the timestamp The fragmented data of the first device locally cached, and the fragmentation data is obtained by fragmenting the full amount of data recorded by the first device.    The device of claim 20, wherein the device further comprises: a second receiving module, configured to receive the first device if the value of the time stamp is equal to or greater than the preset threshold The incremental data sent from the time indicated by the timestamp to the current time.    The device of claim 20, wherein the device further comprises: a triggering module, configured to trigger the first device to send the full amount of data if the value of the timestamp is a specified value; and The third receiving module is configured to receive the fragment data that is sent by the first device and corresponding to the specified value, where the specified value is used to instruct the second device to read the full amount of data to the first device.    The device according to claim 20, wherein the first receiving module comprises: a first receiving unit, configured to receive the specified fragment data corresponding to the index in the local cache of the first device; The index is a value of the timestamp; and the second receiving unit is configured to receive the specified timestamp sent by the first device, where the value of the specified timestamp is obtained by adding 1 to the timestamp .    The apparatus of any one of claims 20 to 23, wherein the full amount of data comprises at least one of: a blocking record for a uniform resource locator, a blocking record for a function variable name.