CN116980435A - Internet of vehicles data synchronization method and device, electronic equipment and storage medium - Google Patents
Internet of vehicles data synchronization method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116980435A CN116980435A CN202311159860.5A CN202311159860A CN116980435A CN 116980435 A CN116980435 A CN 116980435A CN 202311159860 A CN202311159860 A CN 202311159860A CN 116980435 A CN116980435 A CN 116980435A
- Authority
- CN
- China
- Prior art keywords
- data
- networking
- abnormal
- target
- internet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000002159 abnormal effect Effects 0.000 claims abstract description 181
- 230000006855 networking Effects 0.000 claims abstract description 174
- 238000012544 monitoring process Methods 0.000 claims abstract description 132
- 230000008439 repair process Effects 0.000 claims abstract description 34
- 230000008859 change Effects 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 10
- 230000005856 abnormality Effects 0.000 claims description 9
- 238000004140 cleaning Methods 0.000 claims description 5
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 239000004973 liquid crystal related substance Substances 0.000 claims description 4
- 238000012545 processing Methods 0.000 description 15
- 230000003287 optical effect Effects 0.000 description 6
- 238000011084 recovery Methods 0.000 description 6
- 238000012546 transfer Methods 0.000 description 6
- 238000012423 maintenance Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000005067 remediation Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Abstract
The invention discloses a method, a device, electronic equipment and a storage medium for synchronizing internet of vehicles data, wherein the method comprises the following steps: and acquiring first networking data from the target data source based on the data source configuration information in the target data synchronization task, marking the first networking data based on the time window configuration information in the target data synchronization task, acquiring and caching second networking data in a unit time window, and performing data monitoring on the second networking data based on the operation log information corresponding to the unit time window. If the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result; and if the data monitoring result is a normal result, writing and synchronizing the second networking data into the target database. By the technical scheme provided by the embodiment of the invention, the normal synchronization of the Internet of vehicles data is realized, and the abnormal Internet of vehicles data can be repaired in time.
Description
Technical Field
The present invention relates to the field of internet of vehicles, and in particular, to an internet of vehicles data synchronization method, an apparatus, an electronic device, and a storage medium.
Background
Along with the rapid development of the internet of vehicles technology, vehicle-mounted equipment on a vehicle synchronizes vehicle dynamic data in an information network platform in real time through a wireless communication technology, and different functional services are provided in the running of the vehicle.
In the prior art, the one-stop data synchronization system for synchronously solving complex heterogeneous data sources, large data volume, high concurrency data interaction and data synchronization scenes is realized by utilizing the FlinkCDC. However, the real-time synchronization system in the prior art has simple monitoring function, lacks abnormal active alarm, fails data synchronization, cannot repair and the like, and seriously affects daily processing of service data.
Disclosure of Invention
The invention provides a method, a device, electronic equipment and a storage medium for synchronizing internet of vehicles data, which are used for realizing normal synchronization of the internet of vehicles data, monitoring and automatic early warning of the internet of vehicles data, and timely and automatically repairing abnormal internet of vehicles data, so that the workload of internet of vehicles data processing and system operation and maintenance is greatly reduced.
In a first aspect, an embodiment of the present invention provides a method for synchronizing internet of vehicles data, including:
Acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
marking the first networking data based on time window configuration information in the target data synchronization task to obtain second networking data in a unit time window;
caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to obtain a data monitoring result;
if the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result;
and if the data monitoring result is a normal result, writing and synchronizing the second Internet-of-vehicles data into a target database.
In a second aspect, an embodiment of the present invention further provides an internet of vehicles data synchronization device, including:
the first data acquisition module is used for acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
the second data acquisition module is used for marking the first networking data based on the time window configuration information in the target data synchronization task to acquire second networking data in a unit time window;
The data monitoring result acquisition module is used for caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to acquire a data monitoring result;
the abnormality repair module is used for carrying out early warning and abnormality repair on the second networking data based on the target abnormality type in the data monitoring result if the data monitoring result is an abnormality result;
and the second data synchronization module is used for writing and synchronizing the second networking data into a target database if the data monitoring result is a normal result.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes: at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the internet of vehicles data synchronization method provided by any embodiment of the present invention.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where the computer readable storage medium stores computer instructions, where the computer instructions are configured to enable a processor to execute the method for synchronizing internet of vehicles data provided in any embodiment of the present invention.
According to the technical scheme, the first Internet of vehicles data is obtained from the configured target data source based on the data source configuration information in the target data synchronization task. Based on time window configuration information in a target data synchronization task, marking the first networking data according to unit time to obtain second networking data in the unit time window, so that the first networking data is divided into different time segments by using the unit time window, the monitoring and the data recovery of the data can be realized conveniently, the time window provides rich operators, and the coding threshold is reduced. And caching the second networking data in the unit time window, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to acquire a data monitoring result. If the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result; if the data monitoring result is a normal result, writing the second internet of vehicles data into a target database, so that the normal synchronization of the internet of vehicles data is realized, the internet of vehicles data can be monitored, abnormal internet of vehicles data can be automatically repaired in time, and the workload of internet of vehicles data processing and system operation and maintenance is greatly reduced.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a method for synchronizing internet of vehicles according to a first embodiment of the present invention;
FIG. 2 is a flow chart of data synchronization for Internet of vehicles according to a first embodiment of the present invention;
fig. 3 is a flowchart of a method for synchronizing internet of vehicles according to a second embodiment of the present invention;
fig. 4 is a schematic structural diagram of an internet of vehicles data synchronization device according to a third embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device for implementing the internet of vehicles data synchronization method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "target," "current," and the like in the description and claims of the present invention and the above-described drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a method for synchronizing internet of vehicles according to an embodiment of the present invention, where the embodiment is applicable to real-time synchronization of internet of vehicles data. As shown in fig. 1, the method may be performed by a data synchronizer of the internet of vehicles, which may be implemented in the form of hardware and/or software, and which may be configured in an electronic device. As shown in fig. 1, the method specifically includes the following steps:
s110, acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task.
The data source may refer to a database or a database server used for acquiring data. The data source configuration information may refer to a specific data source that needs to perform data synchronization and is configured in the target data synchronization task. The target data sources may refer to various types of data sources that may be connected to through various data connectors in the internet of vehicles system. For example, the type of target data source may be at least one of MySQL, oracle, and log file. Wherein, mySQL connector connects binlog, oracle connector connects LogMiner, and log file connector is responsible for connecting log file, etc. The first internet of vehicles data may refer to original internet of vehicles data, and may refer to target data source data after invalid or erroneous data is cleaned out of original data in the target data source.
Specifically, the target data synchronization task may be configured based on the service requirement, and the data synchronization process may be performed by performing the target data synchronization task. And taking the configured data source as a target data source based on the data source configuration information required to be subjected to data synchronization in the target data synchronization task. As shown in fig. 2, the corresponding original internet of vehicles data is obtained from each target data source by using the data connector corresponding to each target data source and directly used as the first internet of vehicles data, or format conversion and data cleaning are performed on the original internet of vehicles data, so as to obtain the processed first internet of vehicles data.
For example, S110 may include: acquiring original Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task; and calling a preconfigured user-defined function, and performing format conversion and data cleaning on the original Internet of vehicles data to obtain first Internet of vehicles data.
The raw internet of vehicles data may refer to raw data obtained from a data source. Custom Functions may refer to custom UDF (User-Defined-Functions) Functions, which may be written to convert data formats.
Specifically, based on the data source configuration information in the target data synchronization task, the data sources needing to be subjected to data synchronization are determined, and the data sources are determined to be target data sources. And acquiring corresponding original Internet of vehicles data from each target data source according to the data connectors corresponding to each target data source. And calling a preconfigured user-defined function, converting the original internet of vehicles data in different formats into a target data format, and cleaning invalid or wrong data from the original internet of vehicles data to obtain first internet of vehicles data.
And S120, marking the first networking data based on the time window configuration information in the target data synchronization task, and obtaining second networking data in a unit time window.
The time window configuration information may refer to unit time length information for dividing different data. The second networking data may refer to the first networking data obtained per unit time of each data channel of the time window. The unit time window may refer to a window in which the obtained data is divided by unit time.
Specifically, the first internet of vehicles data is obtained through window data channels corresponding to the first internet of vehicles data. And marking each first networking data according to unit time based on the time window configuration information in the target data synchronization task, and taking the marked first networking data in the unit time as second networking data to obtain second networking data in the unit time window.
And S130, caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to obtain a data monitoring result.
The running log information may refer to information such as a start identifier and an end identifier of data in a unit time recorded in a unit time window, and a data amount of second internet data.
Specifically, the second networking data acquired by different data source channels are read, the second networking data are cached, log information of unit time windows corresponding to different data channels is collected, the log information is aggregated and analyzed based on the running log information corresponding to the unit time windows, various indexes in the running log information are monitored, the second networking data are monitored through the change trend of the monitored indexes, and the data monitoring result is acquired, so that when the second networking data are abnormal, early warning is timely and automatic, and abnormal data are repaired. The unit time window divides the received first Internet of vehicles data into different time slices, so that the monitoring and data recovery of the first Internet of vehicles data are convenient to realize. The time window provides rich operators at the same time, and the coding threshold is reduced.
And S140, if the data monitoring result is an abnormal result, performing early warning and abnormal repair on the second networking data based on the target abnormal type in the data monitoring result.
The abnormal result may be an abnormal result of the running log information monitoring index of the unit time window corresponding to the second internet data. The target exception types include, but are not limited to, data synchronization exceptions, monitoring traffic delay exceptions, monitoring node states and failover exceptions, data definition (Data Definition Language, DDL) statement execution exceptions, monitoring heartbeat signal reception exceptions, change trend exceptions of various indexes of a unit time window, and the like.
Specifically, as shown in fig. 2, if the data monitoring result is an abnormal result, the abnormal result is analyzed and judged according to a preset processing mode, it is determined that the data monitoring result is a target abnormal type corresponding to the abnormal result, the second internet data is pre-warned based on pre-warning modes corresponding to different target abnormal types, and the second internet data is subjected to abnormal repair based on abnormal repair modes corresponding to different target abnormal types. As shown in fig. 2, after the abnormal repair processing is performed on the second internet data, the repaired second internet data is subjected to data monitoring again, and if the data monitoring result is normal, the repaired second internet data is subjected to data synchronization.
Exemplary, "early warning and anomaly repair for second networking data based on the target anomaly type in the data monitoring result" in S140 may include: determining a target early warning mode and a target repairing mode corresponding to the second networking data based on the target abnormal type, the corresponding relation between the pre-configured abnormal type and the early warning mode and the corresponding relation between the abnormal type and the repairing mode in the data monitoring result; and carrying out abnormal early warning on the second networking data based on the target early warning mode, and carrying out abnormal repair on the second networking data based on the target repair mode.
Specifically, based on the target anomaly type in the data monitoring result, the corresponding relation between the preconfigured anomaly type and the early warning mode and the corresponding relation between the anomaly type and the repair mode, the target anomaly type in the data monitoring result is matched with the corresponding relation between the preconfigured anomaly type and the early warning mode, and the target early warning mode corresponding to the second networking data is determined. And matching the corresponding relation between the target abnormal type in the data monitoring result and the pre-configured abnormal type and the repairing mode, and determining the target repairing mode corresponding to the second networking data. The target repair method may refer to a method of restarting a service, recovering lost data, and the like. And carrying out abnormal early warning on the second networking data based on the target early warning mode, and carrying out abnormal repair on the second networking data based on the target repair mode.
And S150, if the data monitoring result is a normal result, writing and synchronizing the second networking data into the target database.
The target database may refer to a relational database and a non-relational database. It should be noted that the target database is also one of the target data sources.
Specifically, if the monitoring results corresponding to the monitoring indexes of the second networking data are all normal results, determining that the data monitoring results of the second networking data are normal results, and writing the second networking data with the monitoring results being the normal results into the target database for synchronization. Therefore, data interaction and data synchronization among all data sources are realized, the problem of failure in data synchronization of the Internet of vehicles caused by abnormality of the data of the Internet of vehicles is avoided, the probability of data abnormal suspension in the data synchronization process of the Internet of vehicles is greatly reduced, and the probability of successful data synchronization of the Internet of vehicles is improved.
According to the technical scheme, the first Internet of vehicles data is obtained from the configured target data source based on the data source configuration information in the target data synchronization task. Based on time window configuration information in a target data synchronization task, marking the first networking data according to unit time to obtain second networking data in the unit time window, so that the first networking data is divided into different time segments by using the unit time window, the monitoring and the data recovery of the data can be realized conveniently, the time window provides rich operators, and the coding threshold is reduced. And caching the second networking data in the unit time window, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to acquire a data monitoring result. If the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result; if the data monitoring result is a normal result, writing the second internet of vehicles data into the target database, so that the normal synchronization of the internet of vehicles data is realized, the internet of vehicles data can be monitored, abnormal internet of vehicles data can be automatically repaired in time, and the workload of internet of vehicles data processing and system operation and maintenance is greatly reduced.
On the basis of the above technical solution, after S150, the method further includes: determining check codes corresponding to the second networking data synchronized to the target database; if the check code corresponding to each second networking data is detected to be unequal to the check code corresponding to the first networking data, the first networking data is read again and data writing is carried out based on the full snapshot or the incremental snapshot corresponding to the first networking data.
The check code may be an 8-bit 16-ary number check code generated by cyclic redundancy check (Cyclic Redundancy Check), and may be represented by CRC 32.
Specifically, after writing and synchronizing the second networking data into the target database, when a data change event of the target database is sent to Kafka as a message, calculating check codes of each second networking data synchronized into the target database and corresponding first networking data, comparing the second networking data check codes with the corresponding first networking data check codes, and if the check codes corresponding to each second networking data are detected to be not equal to the check codes corresponding to the first networking data, re-reading the first networking data and writing the data based on the first networking data backed up by the full snapshot or the incremental snapshot corresponding to the first networking data. The change condition of the internet of vehicles data is recorded through the real-time snapshot and the snapshot check points, and the data loss and damage are avoided. Meanwhile, the latest snapshot point is kept through a heartbeat mechanism, the binlog expiration is avoided, and the consistency and the integrity of data are ensured; through a snapshot mechanism and a heartbeat mechanism, data synchronization and remote disaster recovery among different target databases can be realized. The data synchronization can use a time stamp-based or incremental backup mode to ensure the real-time synchronization of the data. It should be noted that, the full snapshot refers to performing a complete backup on all the first internet-of-vehicles data, and is generally performed at an initial stage of data synchronization or on a certain key node, so as to ensure the integrity and consistency of the data; the full snapshot requires that all data be retrieved from the source database, potentially creating network and performance bottlenecks. It should be noted that when the original internet of vehicles data is updated or lost, the use of incremental snapshots may cause data inconsistency problems. Thus, in some cases, we may need to re-generate the full snapshot to ensure accuracy of the data. Incremental snapshots refer to backing up only the first networking data that has changed, typically performed periodically, and recording changes to the data by adding snapshot checkpoints. The method can reduce the size and time of the backup data and improve the backup and recovery efficiency; incremental snapshots are implemented by recording the message offset (offset) of Kafka. Each message in Kafka has a unique offset that indicates the location of the message within a particular partition. When we need to acquire the incremental data, we need to record the offset of the last message and then start sending a new message after the offset. The incremental snapshot records only unprocessed messages, as opposed to the full snapshot, thus making storage space smaller and enabling faster processing of large-scale data streams.
Example two
Fig. 3 is a flowchart of a method for synchronizing internet of vehicles data according to a second embodiment of the present invention, where the step of "based on the log information corresponding to the unit time window, performing data monitoring on the second internet of vehicles data, and obtaining the data monitoring result" is optimized on the basis of the above embodiments. Wherein the explanation of the same or corresponding terms as those of the above embodiments is not repeated herein.
Referring to fig. 3, another method for synchronizing internet of vehicles data provided in this embodiment specifically includes the following steps:
s210, acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task.
S220, marking the first networking data based on the time window configuration information in the target data synchronization task, and obtaining second networking data in a unit time window.
S230, caching the second networking data, and determining a monitoring index corresponding to the second networking data based on the operation log information corresponding to the unit time window.
Specifically, the second networking data is cached, running log information corresponding to a unit time window of the data channel is collected, running log information corresponding to the window in the unit time window is aggregated, various monitoring indexes are extracted from the aggregated log information, and various monitoring indexes extracted after the running log information corresponding to the unit time window is aggregated are determined to be the monitoring indexes corresponding to the second networking data.
Exemplary monitoring metrics include, but are not limited to: the data amount is read, the data amount is written, the read rate, the write rate, the fail-over state, the data definition statement execution state, and the heartbeat signal receiving state.
S240, detecting whether the second networking data meets the abnormal conditions corresponding to each abnormal type or not based on the monitoring indexes.
Specifically, based on the monitoring indexes, detecting whether the variation trend of each monitoring index of the second networking data is abnormal, and when the variation trend of the monitoring index is abnormal, the abnormal type of the second networking data is the abnormal type corresponding to the abnormal monitoring index, and the second networking data meets the abnormal condition corresponding to the abnormal type. When the change trend of the monitoring index is normal, the second networking data does not meet the abnormal conditions corresponding to each abnormal type.
For example, S240 may include: detecting whether the second networking data meets an abnormal condition corresponding to the data integrity abnormal type based on the read data amount, the write data amount, the read rate and the write rate; detecting whether the second networking data meets an abnormal condition corresponding to the service delay abnormal type or not based on the read data quantity and the read rate; detecting whether the second networking data meets an abnormal condition corresponding to the failover abnormal type based on the failover state; detecting whether the second networking data meets an abnormal condition corresponding to the data change abnormal type or not based on the data definition statement execution state; based on the heartbeat signal receiving state, whether the second networking data meets an abnormal condition corresponding to the service operation abnormal type or not is detected.
Specifically, based on the read data amount, the write data amount, the read rate and the write rate, whether the change trend of the read data amount, the write data amount, the read rate and the write rate of the second networking data is abnormal is judged, if the change trend is abnormal, the second networking data meets the abnormal condition corresponding to the abnormal type of the data integrity, and if the change trend is normal, the second networking data does not meet the abnormal condition corresponding to the abnormal type of the data integrity. Based on the read data volume and the read rate, judging whether the change trend of the read data volume and the read rate of the second internet is abnormal, if the change trend is abnormal, the second internet data meets the abnormal condition corresponding to the service delay abnormal type, and if the change trend is normal, the second internet data does not meet the abnormal condition corresponding to the service delay abnormal type. Based on the fault transfer state, whether the system automatically transfers the workload to the backup node to continue operation is detected, if yes, the second networking data meets the abnormal condition corresponding to the fault transfer abnormal type, and if not, the second networking data does not meet the abnormal condition corresponding to the fault transfer abnormal type. Based on the execution state of the data definition statement, judging whether the execution situation of the DDL statement has an abnormal situation, if the execution situation of the DDL statement has an abnormal situation, judging whether the second networking data meets the abnormal condition corresponding to the data change abnormal type, and if the execution situation of the DDL statement is normal, judging that the second networking data does not meet the abnormal condition corresponding to the data change abnormal type. Based on the heartbeat signal receiving state, judging whether the receiving condition of the heartbeat signal is abnormal, if the receiving condition of the heartbeat signal is abnormal, the second networking data meets the abnormal condition corresponding to the service operation abnormal type, and if the receiving condition of the heartbeat signal is normal, the second networking data does not meet the abnormal condition corresponding to the service operation abnormal type.
S250, if the second networking data meets the abnormal condition corresponding to at least one abnormal type, determining that the data monitoring result is an abnormal result.
Specifically, if the second networking data meets the abnormal condition corresponding to at least one of the data integrity abnormal type, the service delay abnormal type, the fault transfer abnormal type, the data change abnormal type and the service operation abnormal type, determining that the data monitoring result of the second networking data is an abnormal result.
And S260, if the second networking data do not meet the abnormal conditions corresponding to each abnormal type, determining that the data monitoring result is a normal result.
Specifically, if the second networking data does not meet the abnormal condition corresponding to any one of the data integrity abnormal type, the service delay abnormal type, the fault transfer abnormal type, the data change abnormal type and the service operation abnormal type, determining that the data monitoring result of the second networking data is a normal result.
S270, if the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result.
And S280, if the data monitoring result is a normal result, writing and synchronizing the second networking data into the target database.
According to the technical scheme, the log information in the unit time window is aggregated through the running log information corresponding to the unit time window, so that the second networking data is monitored, and the monitoring index corresponding to the second networking data is determined from the aggregated log information. Detecting whether the second networking data meets the abnormal conditions corresponding to each abnormal type or not in real time based on the monitoring indexes; if the second networking data meets the abnormal condition corresponding to at least one abnormal type, determining that the data monitoring result is an abnormal result; if the second internet of vehicles data does not meet the abnormal conditions corresponding to each abnormal type, the data monitoring result is determined to be a normal result, so that accurate monitoring of the real-time synchronous process of the internet of vehicles data is realized, abnormal internet of vehicles data is automatically repaired in time, and the workload of internet of vehicles data processing and system operation and maintenance is greatly reduced.
Example III
Fig. 4 is a schematic structural diagram of a data synchronization device for internet of vehicles according to a third embodiment of the present invention. As shown in fig. 4, the apparatus includes: a first data acquisition module 310, a second data acquisition module 320, a data monitoring result acquisition module 330, an anomaly remediation module 340, and a second data synchronization module 350.
The first data obtaining module 310 is configured to obtain first internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
a second data obtaining module 320, configured to mark the first networking data based on the time window configuration information in the target data synchronization task, to obtain second networking data in a unit time window;
the data monitoring result obtaining module 330 is configured to cache the second networking data, and perform data monitoring on the second networking data based on the operation log information corresponding to the unit time window, so as to obtain a data monitoring result;
the anomaly repair module 340 is configured to, if the data monitoring result is an anomaly result, perform early warning and anomaly repair on the second networking data based on a target anomaly type in the data monitoring result;
and the second data synchronization module 350 is configured to perform write synchronization on the second networking data to a target database if the data monitoring result is a normal result.
According to the technical scheme of the embodiment, first Internet of vehicles data is obtained from the configured target data source based on the data source configuration information in the target data synchronization task. Based on time window configuration information in a target data synchronization task, marking the first networking data according to unit time to obtain second networking data in the unit time window, so that the first networking data is divided into different time segments by using the unit time window, the monitoring and the data recovery of the data can be realized conveniently, the time window provides rich operators, and the coding threshold is reduced. And caching the second networking data in the unit time window, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to acquire a data monitoring result. If the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result; if the data monitoring result is a normal result, writing the second internet of vehicles data into a target database, so that the normal synchronization of the internet of vehicles data is realized, the internet of vehicles data can be monitored, abnormal internet of vehicles data can be automatically repaired in time, and the workload of internet of vehicles data processing and system operation and maintenance is greatly reduced.
Optionally, the first data acquisition module 310 includes:
the original data acquisition unit is used for acquiring original Internet of vehicles data from a target data source based on the data source configuration information in the target data synchronization task;
the first data acquisition unit is used for calling a preconfigured user-defined function, performing format conversion and data cleaning on the original Internet of vehicles data, and obtaining first Internet of vehicles data.
Optionally, the data monitoring result obtaining module 330 includes:
the monitoring index determining unit is used for determining a monitoring index corresponding to the second networking data based on the operation log information corresponding to the unit time window;
the abnormal condition monitoring unit is used for detecting whether the second networking data meets the abnormal conditions corresponding to each abnormal type or not based on the monitoring indexes;
the abnormal result determining unit is used for determining that the data monitoring result is an abnormal result if the second networking data meets the abnormal condition corresponding to at least one abnormal type;
and the normal result determining unit is used for determining that the data monitoring result is a normal result if the second networking data does not meet the abnormal conditions corresponding to each abnormal type.
Optionally, the monitoring index includes: the data amount is read, the data amount is written, the read rate, the write rate, the fail-over state, the data definition statement execution state, and the heartbeat signal receiving state.
Optionally, the abnormal condition monitoring unit is specifically configured to: detecting whether the second networking data meets an abnormal condition corresponding to a data integrity abnormal type based on the read data amount, the write data amount, the read rate and the write rate; detecting whether the second networking data meets an abnormal condition corresponding to a service delay abnormal type or not based on the read data quantity and the read rate; detecting whether the second networking data meets an abnormal condition corresponding to a failover abnormal type based on the failover state; detecting whether the second internet-of-vehicle data meets an abnormal condition corresponding to a data change abnormal type or not based on the data definition statement execution state; and detecting whether the second networking data meets an abnormal condition corresponding to a service operation abnormal type or not based on the heartbeat signal receiving state.
Optionally, the exception repair module 340 includes:
the early warning mode and repair mode determining unit is used for determining a target early warning mode and a target repair mode corresponding to the second networking data based on the target abnormal type in the data monitoring result, the corresponding relation between the pre-configured abnormal type and the early warning mode and the corresponding relation between the abnormal type and the repair mode;
And the early warning and repairing unit is used for carrying out abnormal early warning on the second networking data based on the target early warning mode and carrying out abnormal repairing on the second networking data based on the target repairing mode.
Optionally, the apparatus further comprises:
the verification code determining unit is used for determining the verification code corresponding to each second networking data synchronized into the target database after writing and synchronizing the second networking data into the target database;
and the rewriting unit is used for reading the first Internet of vehicles data again and writing the data based on the full snapshot or the incremental snapshot corresponding to the first Internet of vehicles data if detecting that the check code corresponding to each second Internet of vehicles data is not equal to the check code corresponding to the first Internet of vehicles data.
The Internet of vehicles data synchronization device provided by the embodiment of the invention can execute the Internet of vehicles data synchronization method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 5 shows a schematic diagram of the structure of an electronic device 12 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as desktop computers, workstations, servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile equipment, such as personal digital processing, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing equipment. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 5, the electronic device 12 is in the form of a general purpose computing device. Components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, a bus 18 that connects the various system components, including the system memory 28 and the processing units 16.
Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
Electronic device 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by electronic device 12 and includes both volatile and nonvolatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. The electronic device 12 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 5, commonly referred to as a "hard disk drive"). Although not shown in fig. 5, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each drive may be coupled to bus 18 through one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored in, for example, system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.
The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and/or any devices (e.g., network card, modem, etc.) that enable the electronic device 12 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 22. Also, the electronic device 12 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 over the bus 18. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 12, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, for example, implementing a method for synchronizing data of the internet of vehicles according to the embodiment of the present invention, the method includes:
acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
marking the first networking data based on time window configuration information in the target data synchronization task to obtain second networking data in a unit time window;
caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to obtain a data monitoring result;
if the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result;
and if the data monitoring result is a normal result, writing and synchronizing the second Internet-of-vehicles data into a target database.
Of course, those skilled in the art will understand that the processor may also implement the technical solution of the internet of vehicles data synchronization method provided by any embodiment of the present invention.
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the internet of vehicles data synchronization method as provided by any embodiment of the present invention, the method comprising:
acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
marking the first networking data based on time window configuration information in the target data synchronization task to obtain second networking data in a unit time window;
caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to obtain a data monitoring result;
if the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result;
and if the data monitoring result is a normal result, writing and synchronizing the second Internet-of-vehicles data into a target database.
The computer storage media of embodiments of the invention may take the form of any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
It will be appreciated by those of ordinary skill in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be centralized on a single computing device, or distributed over a network of computing devices, or they may alternatively be implemented in program code executable by a computer device, such that they are stored in a memory device and executed by the computing device, or they may be separately fabricated as individual integrated circuit modules, or multiple modules or steps within them may be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (10)
1. The internet of vehicles data synchronization method is characterized by comprising the following steps:
acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
marking the first networking data based on time window configuration information in the target data synchronization task to obtain second networking data in a unit time window;
caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to obtain a data monitoring result;
if the data monitoring result is an abnormal result, early warning and abnormal repair are carried out on the second networking data based on the target abnormal type in the data monitoring result;
and if the data monitoring result is a normal result, writing and synchronizing the second Internet-of-vehicles data into a target database.
2. The method of claim 1, wherein the obtaining the first internet of vehicles data from the target data source based on the data source configuration information in the target data synchronization task comprises:
acquiring original Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
And calling a preconfigured user-defined function, and performing format conversion and data cleaning on the original Internet of vehicles data to obtain first Internet of vehicles data.
3. The method according to claim 1, wherein the performing data monitoring on the second networking data based on the running log information corresponding to the unit time window, and obtaining a data monitoring result, includes:
determining a monitoring index corresponding to the second Internet of vehicles data based on the operation log information corresponding to the unit time window;
detecting whether the second networking data meets abnormal conditions corresponding to each abnormal type or not based on the monitoring indexes;
if the second networking data meets the abnormal condition corresponding to at least one abnormal type, determining that the data monitoring result is an abnormal result;
and if the second networking data do not meet the abnormal conditions corresponding to each abnormal type, determining that the data monitoring result is a normal result.
4. A method according to claim 3, wherein the monitoring criteria comprises: the data amount is read, the data amount is written, the read rate, the write rate, the fail-over state, the data definition statement execution state, and the heartbeat signal receiving state.
5. The method of claim 4, wherein detecting whether the second networking data satisfies an anomaly condition for each anomaly type based on the monitoring metrics comprises:
detecting whether the second networking data meets an abnormal condition corresponding to a data integrity abnormal type based on the read data amount, the write data amount, the read rate and the write rate;
detecting whether the second networking data meets an abnormal condition corresponding to a service delay abnormal type or not based on the read data quantity and the read rate;
detecting whether the second networking data meets an abnormal condition corresponding to a failover abnormal type based on the failover state;
detecting whether the second internet-of-vehicle data meets an abnormal condition corresponding to a data change abnormal type or not based on the data definition statement execution state;
and detecting whether the second networking data meets an abnormal condition corresponding to a service operation abnormal type or not based on the heartbeat signal receiving state.
6. The method of claim 1, wherein the pre-warning and anomaly repair of the second networking data based on the target anomaly type in the data monitoring results comprises:
Determining a target early warning mode and a target repairing mode corresponding to the second networking data based on the target abnormal type, the corresponding relation between the pre-configured abnormal type and the early warning mode and the corresponding relation between the abnormal type and the repairing mode in the data monitoring result;
and carrying out abnormal early warning on the second networking data based on the target early warning mode, and carrying out abnormal repair on the second networking data based on the target repair mode.
7. The method of claim 1, further comprising, after write synchronizing the second networking data into a target database:
determining check codes corresponding to the second networking data synchronized to the target database;
if the check code corresponding to each second networking data is detected to be unequal to the check code corresponding to the first networking data, the first networking data is read again and data writing is carried out based on the full-quantity snapshot or the increment snapshot corresponding to the first networking data.
8. An internet of vehicles data synchronization device, comprising:
the first data acquisition module is used for acquiring first Internet of vehicles data from a target data source based on data source configuration information in a target data synchronization task;
The second data acquisition module is used for marking the first networking data based on the time window configuration information in the target data synchronization task to acquire second networking data in a unit time window;
the data monitoring result acquisition module is used for caching the second networking data, and carrying out data monitoring on the second networking data based on the operation log information corresponding to the unit time window to acquire a data monitoring result;
the abnormality repair module is used for carrying out early warning and abnormality repair on the second networking data based on the target abnormality type in the data monitoring result if the data monitoring result is an abnormality result;
and the second data synchronization module is used for writing and synchronizing the second networking data into a target database if the data monitoring result is a normal result.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the internet of vehicles data synchronization method of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the internet of vehicles data synchronization method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311159860.5A CN116980435A (en) | 2023-09-08 | 2023-09-08 | Internet of vehicles data synchronization method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311159860.5A CN116980435A (en) | 2023-09-08 | 2023-09-08 | Internet of vehicles data synchronization method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116980435A true CN116980435A (en) | 2023-10-31 |
Family
ID=88475172
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311159860.5A Pending CN116980435A (en) | 2023-09-08 | 2023-09-08 | Internet of vehicles data synchronization method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116980435A (en) |
-
2023
- 2023-09-08 CN CN202311159860.5A patent/CN116980435A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107908494B (en) | Abnormal event processing method and device, electronic equipment and storage medium | |
| Fu et al. | Digging deeper into cluster system logs for failure prediction and root cause diagnosis | |
| CN111339175B (en) | Data processing method, device, electronic equipment and readable storage medium | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| US9342390B2 (en) | Cluster management in a shared nothing cluster | |
| CN112559475B (en) | Data real-time capturing and transmitting method and system | |
| CN111581056B (en) | Software engineering database maintenance and early warning system based on artificial intelligence | |
| CN111338834B (en) | Data storage method and device | |
| CN114048217A (en) | Incremental data synchronization method and device, electronic equipment and storage medium | |
| CN112069264A (en) | Heterogeneous data source acquisition method and device, electronic equipment and storage medium | |
| CN111753012A (en) | Data synchronization method, device, equipment and storage medium | |
| CN114925140A (en) | Data synchronization system | |
| CN111198916B (en) | Data transmission method and device, electronic equipment and storage medium | |
| CN113377863B (en) | Data synchronization method and device, electronic equipment and computer readable storage medium | |
| CN109885431B (en) | Method and apparatus for backing up data | |
| CN116980435A (en) | Internet of vehicles data synchronization method and device, electronic equipment and storage medium | |
| CN111782431A (en) | Exception processing method, exception processing device, terminal and storage medium | |
| CN111324514A (en) | System abnormity monitoring method, device, equipment and storage medium | |
| CN111324516A (en) | Method and device for automatically recording abnormal event, storage medium and electronic equipment | |
| CN112818204B (en) | Service processing method, device, equipment and storage medium | |
| CN114996057A (en) | Data backup method and device, electronic equipment and computer readable storage medium | |
| CN111159237B (en) | System data distribution method and device, storage medium and electronic equipment | |
| CN109918231B (en) | Data reforming abnormity repairing method, device, equipment and storage medium | |
| CN112463779A (en) | Data processing method, device, equipment and storage medium | |
| CN113760920A (en) | Data synchronization method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |