CN113300911A - Method, device, equipment and readable medium for processing multi-node data transfer error - Google Patents
Method, device, equipment and readable medium for processing multi-node data transfer error Download PDFInfo
- Publication number
- CN113300911A CN113300911A CN202110527881.2A CN202110527881A CN113300911A CN 113300911 A CN113300911 A CN 113300911A CN 202110527881 A CN202110527881 A CN 202110527881A CN 113300911 A CN113300911 A CN 113300911A
- Authority
- CN
- China
- Prior art keywords
- data
- name
- responding
- node
- judging whether
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012546 transfer Methods 0.000 title claims abstract description 16
- 238000012545 processing Methods 0.000 title abstract description 9
- 238000004590 computer program Methods 0.000 claims description 11
- 230000004044 response Effects 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 abstract description 12
- 238000012216 screening Methods 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 241001269238 Data Species 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Quality & Reliability (AREA)
- Environmental & Geological Engineering (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a device, equipment and a readable medium for processing multi-node data transfer errors, wherein the method comprises the following steps: acquiring source data in the nodes, and converting the source data into a table according to the attribute information; judging whether repeated data exists in the data in the table or not based on the node attributes; and responding to the existence of repeated data in the table, adding the repeated data into the feedback list and alarming. By using the scheme of the invention, the accuracy of data transmission can be increased, the working efficiency of data screening and debugging is increased, the risk of data transmission errors is reduced, the experience of a user is improved, and the product competitiveness is improved.
Description
Technical Field
The field relates to the field of computers, and more particularly to a method, apparatus, device and readable medium for handling multi-node data transfer errors.
Background
The modern society is a widely spread society of data and information, and when data is transmitted, the source of the data mostly comes from third-party companies or individuals, so that the data quality of the data source has great uncertainty. At the application level, data is typically expressed, stored, and transmitted in json or XML, among other forms. Taking weather information as an example, more than 2000 cities exist in China, and when weather data is released at a certain weather information website, the weather information of each city contains information such as a message ID, a city name, temperature and the like. According to normal logic, each CITY has a unique CITY ID number (CITY _ ID), so that the condition of CITY weather information error generally cannot occur, but the weather information provided by a third-party website is lost, repeated and wrong in practice.
The data loss mainly reflects that the weather information of the whole city of China transmitted every time is not completely provided, the weather information of some cities can be lost in some time periods, and particularly, the weather information of small cities is abundant. The data repetition mainly reflects the same city, and can be transmitted for many times in weather information data of a finished national city. The data errors are mainly caused by the fact that multiple cities in China have the same name, for example, 2 jiulong, hong kong jiulong and Sichuan jiulong exist in China, although the names are the same, the phenomenon that weather data are exchanged wrongly often occurs.
Loss, duplication, and error of data are widespread in socially mass-production environments, such as: news and short videos also suffer from the above problem, and a piece of news includes an index ID, a message ID, a news headline, a news digest, news content, and the like. Old news can change a new index ID or message ID, under the unchangeable condition of news title, carry out 2 times propelling movement and lead to news data repetition, the user receives not filterable information and can reduce reading experience. Similar push data repetition problems also exist in recommendation algorithms in the short video domain.
At present, data processing is currently performed in data transmission, because data sources mostly come from data collection, problems of data errors, data loss or data duplication and the like can occur due to human or other factors in multi-node transmission, and the problems occur in various industries. However, when the data size cannot reach the magnitude of large data, resources are wasted by processing problems through a heavyweight data processing system, data abnormality is often ignored, error correction is mostly performed in a manual mode, and it is difficult to quickly and effectively check out error, repeated or lost data from a large amount of data.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a device, and a readable medium for processing a multi-node data transfer error, which can increase accuracy of data transfer, increase work efficiency of data screening and debugging, reduce risk of data transfer error, improve user experience, and improve product competitiveness.
In view of the above object, an aspect of embodiments of the present invention provides a method of handling a multi-node data transfer error, including the steps of:
acquiring source data in the nodes, and converting the source data into a table according to the attribute information;
judging whether repeated data exists in the data in the table or not based on the node attributes;
and responding to the existence of repeated data in the table, adding the repeated data into the feedback list and alarming.
According to an embodiment of the present invention, further comprising:
responding to the data in the table without repeated data, and judging whether error data exists in the data in the table;
and responding to the error data in the table, adding the error data into the feedback list and alarming.
According to an embodiment of the present invention, further comprising:
responding to the data in the table without error data, and judging whether the data in the table has new and/or lost data;
and adding the new and/or lost data into the feedback list and alarming in response to the new and/or lost data in the table.
According to one embodiment of the present invention, the attribute information includes an index number, a data name, a data body object, specific information of a data body, and a data source address.
According to an embodiment of the present invention, determining whether duplicate data exists in the data in the table based on the node attribute includes:
responding to the requirement of the current node attribute as data number uniqueness, and judging whether the data number columns in the data in the table have the same data number or not;
responding to the requirement that the attribute of the current node is the uniqueness of the data name, and judging whether the data name columns in the data in the table have the same data name or not;
responding to the uniqueness requirement of the data source address of the current node attribute, and judging whether the data source address columns in the data in the table have the same data source address or not;
responding to the requirement of the current node attribute as data number uniqueness and data name uniqueness, and judging whether the data number columns and/or the data name columns in the data in the table have the same data numbers and/or data names;
and in response to the requirement of the current node attribute as data number uniqueness, data name uniqueness and data source address uniqueness, judging whether the data number column and/or the data name column and/or the data source address column in the data in the table have the same data number and/or data name and/or data source address.
According to one embodiment of the present invention, determining whether there is error data in the table comprises: comparing the data in the table with the data in the table of the last version corresponding to the data in the table;
judging whether the repeated data exists in the data in the table based on the node attributes comprises: responding to weather information of the current node, comparing a data number column representing city attributes in the data in the table with a data number column in the data in the table of the previous version, and marking information with the same index number and different data numbers as error data; and responding to the fact that the current node is the short video information, comparing a data name column and a data number column in the data in the table with a data name column and a data number column in the data in the table of the previous version, and marking the data with the same data name and different data numbers as suspected error data.
According to an embodiment of the present invention, determining whether new and/or missing data exists in the data in the table includes: comparing the data in the table with the data in the table of the last version corresponding to the data in the table;
judging whether the repeated data exists in the data in the table based on the node attributes comprises: responding to weather information of the current node, and comparing a data number column representing the city attribute in the data in the table with a data number column in the data in the table of the previous version; and marking the information corresponding to the data numbers which exist in the table and do not exist in the table of the previous version as newly-added data, and marking the information corresponding to the data numbers which do not exist in the table and exist in the table of the previous version as lost data.
In another aspect of the embodiments of the present invention, there is also provided an apparatus for processing a multi-node data transfer error, the apparatus including:
the conversion module is configured to acquire source data in the nodes and convert the source data into a table according to the attribute information;
the judging module is configured to judge whether the repeated data exists in the data in the table or not based on the node attribute;
and the alarm module is configured to respond to the existence of the repeated data in the table, add the repeated data into the feedback list and alarm.
In another aspect of an embodiment of the present invention, there is also provided a computer apparatus including:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of any of the methods described above.
In another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium storing a computer program, which when executed by a processor implements the steps of any one of the above-mentioned methods.
The invention has the following beneficial technical effects: the method for processing the multi-node data transmission error provided by the embodiment of the invention comprises the steps of obtaining source data in a node, and converting the source data into a table according to attribute information; judging whether repeated data exists in the data in the table or not based on the node attributes; the technical scheme that the repeated data are added into the feedback list and alarm is given out in response to the fact that the repeated data exist in the data in the table can increase accuracy of data transmission, increase working efficiency of data screening and debugging, reduce risks of data transmission errors, improve experience of users and improve product competitiveness.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
FIG. 1 is a schematic flow chart diagram of a method of handling multi-node data transfer errors in accordance with one embodiment of the present invention;
FIG. 2 is a diagram illustrating an apparatus for handling multi-node data transfer errors according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a computer device according to one embodiment of the present invention;
fig. 4 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
In view of the above objects, a first aspect of embodiments of the present invention proposes an embodiment of a method of handling multi-node data transfer errors. Fig. 1 shows a schematic flow diagram of the method.
As shown in fig. 1, the method may include the steps of:
s1 acquires source data in the node and converts the source data into a table according to the attribute information.
The node herein may be defined as a usage scenario including, but not limited to, news, weather information, short video Data, system backup, etc., the source Data is typically in json format or XML format, the attribute information in the source Data includes an index number ID, a Data number (Data _ ID), a Data name (Data _ name), a Data body object (datas), specific information of the Data body (Data _1), and a Data source address (URL), and the table structure is as shown in table 1 below. Due to different usage scenarios, the attribute requirements in the source Data are different, for example, in a usage scenario of weather information, the Data number (Data _ ID) representing the city name is required to be unique, that is, the same Data number cannot exist in the Data in the converted table at the same time.
Table 1 data structure table
| ID | Data_ID | data_name | datas | data_1 | URL |
| 1 | G1.1.001 | A | Data volume object | abcdef_1 | http://www.xxx.1 |
| 2 | G1.1.002 | B | Data volume object | abcdef_2 | http://www.xxx.2 |
| 3 | G1.1.003 | C | Data volume object | abcdef_3 | http://www.xxx.3 |
| 4 | G1.1.004 | D | Data volume object | abcdef_4 | http://www.xxx.4 |
| 5 | G1.1.003 | B | Data volume object | abcdef_5 | http://www.xxx.2 |
| 6 | G1.1.006 | B | Data volume object | abcdef_6 | http://www.xxx.6 |
| 7 | G1.1.007 | B | Data volume object | abcdef_7 | http://www.xxx.7 |
| 8 | G1.1.008 | B | Data volume object | abcdef_8 | http://www.xxx.8 |
S2 determines whether duplicate data exists in the data in the table based on the node attributes.
Firstly, judging the attribute of a current node, if the attribute of the current node requires uniqueness of a data number, judging whether a data number column in data in a table has the same data number, if the same data number exists, the data in the current table has repeated data, if the attribute of the current node requires uniqueness of a data name, judging whether a data name column in the data in the table has the same data name, if the same data name exists, the data in the current table has repeated data, if the attribute requires uniqueness of a data source address, judging whether a data source address column in the data in the table has the same data source address, if the same data source address exists, the data in the current table has repeated data, and if the attribute requires uniqueness of the data number and the uniqueness of the data name, judging whether the same data number and/or data name exist in a data number column and/or a data name column in the data in the table, if the same data number or the same data name exists, then the data in the current table has repeated data, if the current node attribute is required to be data number uniqueness, data name uniqueness and data source address uniqueness, judging whether the same data number and/or data name and/or data source address exist in the data number column and/or data name column and/or data source address column in the data in the table, if the same data number or the same data name exists or the same data source address exists, then the data in the current table has repeated data, and so on.
And S3, responding to the existence of repeated data in the table, adding the repeated data into the feedback list and alarming.
The same data can be deleted when the same data can be found, and the same data can be marked in the table.
By the technical scheme, the accuracy of data transmission can be improved, the working efficiency of data screening and debugging is improved, the risk of data transmission errors is reduced, the experience of a user is improved, and the product competitiveness is improved.
In a preferred embodiment of the present invention, the method further comprises:
responding to the data in the table without repeated data, and judging whether error data exists in the data in the table;
and responding to the error data in the table, adding the error data into the feedback list and alarming.
If the table does not have repeated data, the data in the table needs to be judged to be error data, the data in the table needs to be compared with the data in the table of the previous version corresponding to the data in the table, if the current node is weather information, the data number representing the city name in the weather information is required to be invariable, namely the data number in the table of each version needs to be completely the same, the data number column representing the city attribute in the data in the table is compared with the data number column in the data in the table of the previous version, and information with the same index number and different data numbers is marked as error data, for example, the city with the index number of 3 in the table of the previous version is Beijing, and the city with the index number of 3 in the current table is Shanghai, the error data is considered to appear. If the current node is short video information, comparing a data name column and a data number column in data in a table with a data name column and a data number column in data in a table of a previous version, and marking the data with the same data name and different data numbers as suspected error data.
In a preferred embodiment of the present invention, the method further comprises:
responding to the data in the table without error data, and judging whether the data in the table has new and/or lost data;
and adding the new and/or lost data into the feedback list and alarming in response to the new and/or lost data in the table. If the table has no repeated data or error data, the data in the table needs to be judged to be newly added or lost, the data in the table needs to be compared with the data in the table of the previous version corresponding to the data in the table, if the current node is weather information, a data number column representing city attributes in the data in the table is compared with a data number column in the data in the table of the previous version, information corresponding to the data number existing in the table and the data number not existing in the table of the previous version is marked as newly added data, and information corresponding to the data number not existing in the table and the data number existing in the table of the previous version is marked as lost data. If the current node is weather information, the data number representing the city name in the weather information is basically fixed, that is, the data number in the table of each version needs to be completely the same, for example, the table of the previous version does not have the data number representing beijing, and the current table has the data number, the information corresponding to the data number is new data, and if the table of the previous version has the data number representing beijing and the current table does not have the data number, the information corresponding to the data number is lost data.
In a preferred embodiment of the present invention, the attribute information includes an index number, a data name, a data body object, specific information of a data body, and a data source address. The index number (ID) is used to distinguish data. In some embodiments, the Data number (Data _ ID) is an identification code that is unique to the Data throughout the project, and no duplication is allowed in the same project. In some embodiments, the data name (data _ name) is a digest description of the data, requiring uniqueness. Data volume objects (datas) are detailed information of data, and are mainly used for data content transmission to users and processors of multiple data nodes. The specific information of the data body is repeatable data. The data source address (URL) is an additional information or media data source address of the data body, which is generally from multiple data sources, such as weather data from various weather stations throughout the country, news data from various news media or reporters throughout the country, short video data from various short video providing organizations, etc.
In a preferred embodiment of the present invention, determining whether duplicate data exists in the data in the table based on the node attribute includes:
responding to the requirement of the current node attribute as data number uniqueness, and judging whether the data number columns in the data in the table have the same data number or not;
responding to the requirement that the attribute of the current node is the uniqueness of the data name, and judging whether the data name columns in the data in the table have the same data name or not;
responding to the uniqueness requirement of the data source address of the current node attribute, and judging whether the data source address columns in the data in the table have the same data source address or not;
responding to the requirement of the current node attribute as data number uniqueness and data name uniqueness, and judging whether the data number columns and/or the data name columns in the data in the table have the same data numbers and/or data names;
and in response to the requirement of the current node attribute as data number uniqueness, data name uniqueness and data source address uniqueness, judging whether the data number column and/or the data name column and/or the data source address column in the data in the table have the same data number and/or data name and/or data source address. Firstly, the attribute of the current node needs to be judged, and the data uniqueness required by the node attribute is different, for example, if the current node is in a weather condition, the data number of a city represented by the node attribute requirement of the weather condition is unique, whether the same data number exists in the data number column in the table is judged, and if the same data number exists, the data number column is marked and an alarm is given.
In a preferred embodiment of the present invention, determining whether there is error data in the table comprises: comparing the data in the table with the data in the table of the last version corresponding to the data in the table;
judging whether the repeated data exists in the data in the table based on the node attributes comprises: responding to weather information of the current node, comparing a data number column representing city attributes in the data in the table with a data number column in the data in the table of the previous version, and marking information with the same index number and different data numbers as error data; and responding to the fact that the current node is the short video information, comparing a data name column and a data number column in the data in the table with a data name column and a data number column in the data in the table of the previous version, and marking the data with the same data name and different data numbers as suspected error data. If the table does not have repeated data, the data in the table needs to be judged to be error data, the data in the table needs to be compared with the data in the table of the previous version corresponding to the data in the table, if the current node is weather information, the data number representing the city name in the weather information is required to be invariable, namely the data number in the table of each version needs to be completely the same, the data number column representing the city attribute in the data in the table is compared with the data number column in the data in the table of the previous version, and information with the same index number and different data numbers is marked as error data, for example, the city with the index number of 3 in the table of the previous version is Beijing, and the city with the index number of 3 in the current table is Shanghai, the error data is considered to appear. If the current node is short video information, comparing a data name column and a data number column in data in a table with a data name column and a data number column in data in a table of a previous version, and marking the data with the same data name and different data numbers as suspected error data.
In a preferred embodiment of the present invention, the determining whether there is new and/or missing data in the table includes: comparing the data in the table with the data in the table of the last version corresponding to the data in the table;
judging whether the repeated data exists in the data in the table based on the node attributes comprises: responding to weather information of the current node, and comparing a data number column representing the city attribute in the data in the table with a data number column in the data in the table of the previous version; and marking the information corresponding to the data numbers which exist in the table and do not exist in the table of the previous version as newly-added data, and marking the information corresponding to the data numbers which do not exist in the table and exist in the table of the previous version as lost data. If the table has no repeated data or error data, the data in the table needs to be judged to be newly added or lost, the data in the table needs to be compared with the data in the table of the previous version corresponding to the data in the table, if the current node is weather information, a data number column representing city attributes in the data in the table is compared with a data number column in the data in the table of the previous version, information corresponding to the data number existing in the table and the data number not existing in the table of the previous version is marked as newly added data, and information corresponding to the data number not existing in the table and the data number existing in the table of the previous version is marked as lost data. If the current node is weather information, the data number representing the city name in the weather information is basically fixed, that is, the data number in the table of each version needs to be completely the same, for example, the table of the previous version does not have the data number representing beijing, and the current table has the data number, the information corresponding to the data number is new data, and if the table of the previous version has the data number representing beijing and the current table does not have the data number, the information corresponding to the data number is lost data.
By the technical scheme, the accuracy of data transmission can be improved, the working efficiency of data screening and debugging is improved, the risk of data transmission errors is reduced, the experience of a user is improved, and the product competitiveness is improved.
It should be noted that, as will be understood by those skilled in the art, all or part of the processes in the methods of the above embodiments may be implemented by instructing relevant hardware through a computer program, and the above programs may be stored in a computer-readable storage medium, and when executed, the programs may include the processes of the embodiments of the methods as described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Furthermore, the method disclosed according to an embodiment of the present invention may also be implemented as a computer program executed by a CPU, and the computer program may be stored in a computer-readable storage medium. The computer program, when executed by the CPU, performs the above-described functions defined in the method disclosed in the embodiments of the present invention.
In view of the above object, according to a second aspect of the embodiments of the present invention, there is provided an apparatus for handling multi-node data transfer errors, as shown in fig. 2, the apparatus 200 includes:
the conversion module is configured to acquire source data in the nodes and convert the source data into a table according to the attribute information;
the judging module is configured to judge whether the repeated data exists in the data in the table or not based on the node attribute;
and the alarm module is configured to respond to the existence of the repeated data in the table, add the repeated data into the feedback list and alarm.
In view of the above object, a third aspect of the embodiments of the present invention provides a computer device. Fig. 3 is a schematic diagram of an embodiment of a computer device provided by the present invention. As shown in fig. 3, an embodiment of the present invention includes the following means: at least one processor S21; and a memory S22, the memory S22 storing computer instructions S23 executable on the processor, the instructions when executed by the processor implementing the above method.
In view of the above object, a fourth aspect of the embodiments of the present invention proposes a computer-readable storage medium. FIG. 4 is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. As shown in fig. 4, the computer readable storage medium stores S31 a computer program that, when executed by a processor, performs the method as described above S32.
Furthermore, the methods disclosed according to embodiments of the present invention may also be implemented as a computer program executed by a processor, which may be stored in a computer-readable storage medium. Which when executed by a processor performs the above-described functions defined in the methods disclosed in embodiments of the invention.
Further, the above method steps and system elements may also be implemented using a controller and a computer readable storage medium for storing a computer program for causing the controller to implement the functions of the above steps or elements.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
In one or more exemplary designs, the functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk, blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A method of handling multi-node data transfer errors, comprising the steps of:
acquiring source data in a node, and converting the source data into a table according to attribute information;
judging whether repeated data exists in the data in the table or not based on the node attributes;
and responding to the existence of repeated data in the table, adding the repeated data into a feedback list and alarming.
2. The method of claim 1, further comprising:
responding to the data in the table without repeated data, and judging whether error data exists in the data in the table;
and responding to error data in the table, adding the error data into the feedback list and alarming.
3. The method of claim 2, further comprising:
responding to the data in the table without error data, and judging whether the data in the table has new and/or lost data;
and responding to the data in the table to have new and/or lost data, adding the new and/or lost data into the feedback list, and alarming.
4. The method of claim 1, wherein the attribute information comprises an index number, a data name, a data body object, data body specific information, and a data source address.
5. The method of claim 1, wherein determining whether duplicate data exists in the data in the table based on the node attributes comprises:
responding to the requirement of the current node attribute as data number uniqueness, and judging whether the data number columns in the data in the table have the same data number or not;
responding to the requirement that the current node attribute is data name uniqueness, and judging whether the data name columns in the data in the table have the same data name or not;
responding to the uniqueness requirement of the data source address of the current node attribute, and judging whether the data source address columns in the data in the table have the same data source address or not;
responding to the requirement of the current node attribute as data number uniqueness and data name uniqueness, and judging whether the data number columns and/or the data name columns in the data in the table have the same data numbers and/or data names;
and in response to the requirement of the current node attribute as data number uniqueness, data name uniqueness and data source address uniqueness, judging whether the data number column and/or the data name column and/or the data source address column in the data in the table have the same data number and/or data name and/or data source address.
6. The method of claim 2, wherein determining whether erroneous data exists in the data in the table comprises: comparing the data in the table with the data in the table of the last version corresponding to the data in the table;
judging whether the repeated data exists in the data in the table based on the node attributes comprises: responding to weather information of the current node, comparing a data number column representing city attributes in the data in the table with a data number column in the data in the table of the previous version, and marking information with the same index number and different data numbers as error data; and responding to the fact that the current node is short video information, comparing a data name column and a data number column in the data in the table with a data name column and a data number column in the data in the table of the previous version, and marking the data with the same data name and different data numbers as suspected error data.
7. The method of claim 3, wherein determining whether new and/or missing data exists in the data in the table comprises: comparing the data in the table with the data in the table of the last version corresponding to the data in the table;
judging whether the repeated data exists in the data in the table based on the node attributes comprises: responding to weather information of the current node, and comparing a data number column representing the city attribute in the data in the table with a data number column in the data in the table of the previous version; and marking the information corresponding to the data numbers which exist in the table and do not exist in the table of the previous version as newly-added data, and marking the information corresponding to the data numbers which do not exist in the table and exist in the table of the previous version as lost data.
8. An apparatus for handling multi-node data transfer errors, the apparatus comprising:
the conversion module is configured to acquire source data in the nodes and convert the source data into a table according to the attribute information;
a judging module configured to judge whether there is duplicate data in the table based on the node attribute;
the alarm module is configured to respond to the fact that repeated data exist in the data in the table, add the repeated data to a feedback list and alarm.
9. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110527881.2A CN113300911A (en) | 2021-05-14 | 2021-05-14 | Method, device, equipment and readable medium for processing multi-node data transfer error |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110527881.2A CN113300911A (en) | 2021-05-14 | 2021-05-14 | Method, device, equipment and readable medium for processing multi-node data transfer error |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113300911A true CN113300911A (en) | 2021-08-24 |
Family
ID=77322315
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110527881.2A Pending CN113300911A (en) | 2021-05-14 | 2021-05-14 | Method, device, equipment and readable medium for processing multi-node data transfer error |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113300911A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080189238A1 (en) * | 2007-02-02 | 2008-08-07 | Microsoft Corporation | Detecting and displaying exceptions in tabular data |
| US20150019382A1 (en) * | 2012-10-19 | 2015-01-15 | Rakuten, Inc. | Corpus creation device, corpus creation method and corpus creation program |
| CN109147283A (en) * | 2018-09-06 | 2019-01-04 | 浙江航天长峰科技发展有限公司 | A kind of short message alarm system and method for safety supervision information |
| WO2020041827A1 (en) * | 2018-08-31 | 2020-03-05 | Jaxsta Enterprise Pty Ltd | Data deduplication and data merging |
-
2021
- 2021-05-14 CN CN202110527881.2A patent/CN113300911A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080189238A1 (en) * | 2007-02-02 | 2008-08-07 | Microsoft Corporation | Detecting and displaying exceptions in tabular data |
| US20150019382A1 (en) * | 2012-10-19 | 2015-01-15 | Rakuten, Inc. | Corpus creation device, corpus creation method and corpus creation program |
| WO2020041827A1 (en) * | 2018-08-31 | 2020-03-05 | Jaxsta Enterprise Pty Ltd | Data deduplication and data merging |
| CN109147283A (en) * | 2018-09-06 | 2019-01-04 | 浙江航天长峰科技发展有限公司 | A kind of short message alarm system and method for safety supervision information |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20080098001A1 (en) | Techniques for efficient loading of binary xml data | |
| CN109104405B (en) | Binary protocol encoding and decoding method and device | |
| CN104794123A (en) | Method and device for establishing NoSQL database index for semi-structured data | |
| CN108733317B (en) | Data storage method and device | |
| KR20190020105A (en) | Method and device for distributing streaming data | |
| CN105187258A (en) | Method for analyzing and processing configurable Trap alarm | |
| US20170032052A1 (en) | Graph data processing system that supports automatic data model conversion from resource description framework to property graph | |
| CN101772043A (en) | Office data checking method and device | |
| CN108664546B (en) | XML data structure conversion method and device | |
| US20190384825A1 (en) | Method and device for data protection and computer readable storage medium | |
| CN112395339B (en) | Intersystem data admission verification method, device, computer equipment and storage medium | |
| CN111124883B (en) | Test case library introduction method, system and equipment based on tree form | |
| CN114327493A (en) | Data processing method and device, electronic equipment and computer readable medium | |
| CN103051480B (en) | The storage means of a kind of DN and DN storage device | |
| CN111078773B (en) | Data processing method and device | |
| CN110049133B (en) | Method and device for issuing full amount of DNS zone files | |
| CN113300911A (en) | Method, device, equipment and readable medium for processing multi-node data transfer error | |
| CN115357286B (en) | Program file comparison method and device, electronic equipment and storage medium | |
| CN115329759B (en) | Information processing method, device, equipment and storage medium | |
| CN108804561B (en) | Data synchronization method and device | |
| CN116303628A (en) | Alarm data query method, system and equipment based on elastic search | |
| CN110569089A (en) | Plug-in expansion method, device and storage medium for management platform | |
| US20220156069A1 (en) | Systems and methods for handling macro compatibility for documents at a storage system | |
| CN109491699B (en) | Resource checking method, device, equipment and storage medium of application program | |
| CN114490651A (en) | Data storage method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210824 |