CN116737710A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN116737710A CN116737710A CN202310827031.3A CN202310827031A CN116737710A CN 116737710 A CN116737710 A CN 116737710A CN 202310827031 A CN202310827031 A CN 202310827031A CN 116737710 A CN116737710 A CN 116737710A
- Authority
- CN
- China
- Prior art keywords
- data
- identifiers
- target data
- application
- repeated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 25
- 238000012545 processing Methods 0.000 claims abstract description 81
- 238000000034 method Methods 0.000 claims abstract description 53
- 230000005540 biological transmission Effects 0.000 claims description 19
- 238000012544 monitoring process Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 9
- 230000008439 repair process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 2
- 244000118350 Andrographis paniculata Species 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Retry When Errors Occur (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a data processing method, a device and electronic equipment, wherein the method comprises the following steps: after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired; and if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
Description
The patent application of the application is a divisional application of Chinese patent application with the application date of 2019, 8 months and 8 days, the application number of 201910728574.3 and the name of data processing method, device and electronic equipment.
Technical Field
The present application relates to the field of communications technologies, and in particular, to a data processing method, an apparatus, and an electronic device.
Background
In the process of data application, the data needs to be further transmitted to a data application party after being generated, and in the process of transmission, the situation of data loss or data repetition often occurs. When data is repeated, it is typically found when the data is applied; when data is lost due to network reasons, server reasons, or code flaws, difficulties are found and the impact is severe.
The existing solution is mainly to directly optimize a server or a data application system, for example, a data repair function or a data deduplication function is added in the server or the data application system, but the above method can increase time consumption for processing data and reduce timeliness and stability of data application. Therefore, a data processing method is needed to overcome the above technical problems.
Disclosure of Invention
One of the purposes of the present application is to provide a data processing method, so as to solve the problems of low timeliness and stability of the data repairing method in the existing data application.
In order to solve the technical problems, the embodiment of the application is realized as follows:
in a first aspect, a data processing method is provided, including:
after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired;
and if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
In a second aspect, a data processing method is provided, including:
acquiring target data, wherein the target data corresponds to a plurality of first identifiers;
and if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
In a third aspect, a data processing method is provided, including:
sending target data to a data application party, wherein the target data corresponds to a plurality of second identifiers;
and if the retransmission instruction is acquired, retransmitting the target data corresponding to the second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application party.
In a fourth aspect, there is provided a data processing apparatus comprising:
an acquisition unit for acquiring a plurality of first identifications corresponding to target data after the target data is transmitted from a data sender to a data application;
and the processing unit is used for sending a de-duplication instruction to the data application party if repeated identifiers exist in the plurality of first identifiers, wherein the de-duplication instruction comprises the repeated identifiers and is used for performing de-duplication processing on the data corresponding to the repeated identifiers in the target data by the data application party.
In a fifth aspect, there is provided a data processing apparatus comprising:
the data acquisition unit acquires target data, wherein the target data corresponds to a plurality of first identifiers;
a deduplication execution unit, if a deduplication instruction is obtained, repeating identifiers in the plurality of first identifiers included in the deduplication instruction; the corresponding target data is deleted or applied only once.
In a sixth aspect, there is provided a data processing apparatus comprising:
the data transmitting unit is used for transmitting target data to the data application party, wherein the target data corresponds to a plurality of second identifiers;
and a retransmission execution unit configured to, if a retransmission instruction is acquired, retransmit target data corresponding to a second identifier included in the retransmission instruction and not included in the plurality of first identifiers to the data application side.
In a seventh aspect, there is provided an electronic device comprising:
a processor; and
a memory arranged to store computer executable instructions that when executed perform the following operations using the processor:
after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired;
and if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
In an eighth aspect, there is provided an electronic device comprising:
a processor; and
a memory arranged to store computer executable instructions that when executed perform the following operations using the processor:
acquiring target data, wherein the target data corresponds to a plurality of first identifiers;
and if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
In a ninth aspect, there is provided an electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions that when executed perform the following operations using the processor:
sending target data to a data application party, wherein the target data corresponds to a plurality of second identifiers;
and if the retransmission instruction is acquired, retransmitting the target data corresponding to the second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application party.
In a tenth aspect, there is provided a computer readable medium storing one or more programs, which when executed by an electronic device, comprising a plurality of application programs, cause the electronic device to:
after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired;
and if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
In an eleventh aspect, there is provided a computer readable medium storing one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:
acquiring target data, wherein the target data corresponds to a plurality of first identifiers;
and if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
In a twelfth aspect, there is provided a computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to:
sending target data to a data application party, wherein the target data corresponds to a plurality of second identifiers;
and if the retransmission instruction is acquired, retransmitting the target data corresponding to the second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application party.
As can be seen from the above technical solutions of the present application, in the data processing method according to the embodiments of the present application, after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are obtained, whether the on-line real-time data are repeated is determined by judging whether there are repeated identifiers in the plurality of first identifiers, and if so, a deduplication instruction is sent to the data application, so that the data application performs deduplication processing on data corresponding to the repeated identifiers in the target data. The data processing method of the embodiment of the application can monitor and repair the real-time data, reduce the time consumption of processing the data on line, correct the error data without perception and influence the normal operation of the business on line.
Further, in the data processing method according to the embodiment of the present application, before the target data is transmitted from the data sender to the data application, a plurality of second identifiers corresponding to the target data may be obtained, and if the first identifier does not include at least one second identifier of the plurality of second identifiers, a retransmission instruction is sent to the data sender, and the data sender performs retransmission processing on the data corresponding to the second identifier that is not included in the first identifier in the target data. The data processing method of the embodiment of the application can repair the lost error of the online data without affecting the normal operation of the online service.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data processing method according to one embodiment of the application.
Fig. 2 is a schematic flow chart of a data processing method according to another embodiment of the application.
Fig. 3 is a schematic flow chart of a data processing method according to another embodiment of the application.
FIG. 4 is a schematic diagram of a data monitoring repair system according to one embodiment of the application.
Fig. 5 is a schematic flow chart of a data processing method according to a specific embodiment of the application.
Fig. 6 is a schematic structural view of an electronic device according to an embodiment of the present application.
Fig. 7 is a schematic structural view of a data processing apparatus according to an embodiment of the present application.
Fig. 8 is a schematic structural view of a data processing apparatus according to another embodiment of the present application.
Fig. 9 is a schematic structural view of a data processing apparatus according to another embodiment of the present application.
Detailed Description
In order to make the technical solution of the present application better understood by those skilled in the art, the technical solution of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, shall fall within the scope of the application.
The data processing method of the embodiment of the application can be applied to the real-time big data processing process, and real-time data repair is carried out through the independent module, so that the normal operation of on-line business is not affected.
FIG. 1 illustrates a data processing method of one embodiment of the present application. The method of fig. 1 may be performed by a data processing apparatus. As shown in fig. 1, the method includes:
s102, after target data are transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired.
Optionally, in S102, the data sender may be a data acquisition system or a data generation system, and when the data sender is a data acquisition system, the data acquisition system is connected to the data generation system. Specifically, the target data may include a plurality of pieces of data, each piece of data corresponds to one first identifier, and different pieces of data correspond to different first identifiers.
And S104, if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
In the process of transmitting the target data from the data sender to the data application, data errors may occur, and the target data received by the data application is different from the data actually intended to be transmitted by the data sender, for example, there is a case that the data is repeated, etc. And determining whether the first identifiers are repeated or not by verifying whether the first identifiers are the same or not, and if so, indicating that repeated data exist in the target data, wherein the repeated data need to be deduplicated.
Specifically, a deduplication instruction may be sent to the data application party, where the deduplication instruction includes the repeated identifier, and the data application party may intercept or delete data according to the repeated identifier. Therefore, the data application party or the data sender is not required to directly perform data verification, real-time data can be monitored and repaired in an independent monitoring module, the time consumption for processing the data online is reduced, and the normal operation of online business is not influenced.
In an embodiment of the present application, optionally, the method shown in fig. 1 further includes: before target data is transmitted from the data sender to the data application party, a plurality of second identifiers corresponding to the target data are acquired;
and if at least one second identifier in the plurality of second identifiers is not included in the plurality of first identifiers, transmitting a retransmission instruction to the data transmitting party. The retransmission instruction includes a second identifier not included in the plurality of first identifiers, and is used for the data sender to retransmit data corresponding to the second identifier not included in the plurality of first identifiers in the target data.
By comparing the second identifier corresponding to the target data before transmission to the data application side and the first identifier corresponding to the target data after transmission to the data application side, it is possible to determine whether an error occurs in the data after transmission with respect to the target data before transmission. It should be noted that, the plurality of first identifiers are identifiers corresponding to the target data received by the data application party in the preset period. Specifically, if at least one of the plurality of second identifiers is not included in the first identifier, the target data is lost, and the data sender is required to retransmit the lost data. Wherein data corresponding to a second identity that is not included in the plurality of first identities may be retransmitted.
Specifically, in some embodiments, the second identification is generated based on at least one of the following information: data acquisition source information, data generation time information, data number information and data generation mode information. Alternatively, the second identifier may be a uniquely coded identifier code. The data acquisition source may be sls/MetaQ/kafka/flume or the like, and may be represented by S, M, K, F in the second identifier; the data generation method may be generation, repetition generation, loss generation, or the like, and may be represented by creat, repeated, lose in the second identifier. For example, if the second identifier is s_201903262040000_000001_create, the source of the acquisition system is sls, the data generation time is 49 minutes at 20/26/3/2019, the data number is 1, and the data generation mode is generation; if the second identifier is f_201903262040000_000001_repeated, the source of the acquisition system is resume, the data generation time is 49 minutes at 20 days of 2019, 3 months and 26 days, the data number is 1, and the data generation mode is repeated generation.
In some embodiments, the data processing apparatus may further provide a repeated early warning function, and optionally, the method shown in fig. 1 further includes:
and if the target data have repeated data, sending repeated early warning information to the data application party and/or the data sender. Specifically, the repeated pre-warning information includes a repeated identification. By sending the early warning information to the data application party and/or the data sender, the data application party and the data sender can prompt that the repetition condition occurs in the currently transmitted target data, and can determine which data are repeated based on the repeated identification in the early warning information, so that the duplicate removal processing or the system maintenance can be performed.
And after the data application side successfully performs the de-duplication processing, sending repeated early warning cancellation information to the data application side and/or the data sender, wherein the repeated early warning cancellation information comprises repeated identifiers. The successful deduplication process may be that the data application side deletes the repeated data in the target data, or that the data application side only executes one application on the repeated data in the target data, and may send repeated early warning cancellation information again to prompt that the repeated problem has been repaired.
In other embodiments, the data processing apparatus may further provide a loss warning function, and optionally, the method shown in fig. 1 further includes:
and if the target data lose data, sending loss early warning information to the data application party and/or the data sender, wherein the loss early warning information comprises second identifiers which are not included in the plurality of first identifiers.
And after the retransmission processing of the data sender is successful, sending loss early warning cancellation information to the data application party and/or the data sender, wherein the loss early warning cancellation information comprises second identifiers which are not included in the plurality of first identifiers.
The retransmission process may be successful, that is, the data sender retransmits the lost data in the target data, where the lost data is the data corresponding to the second identifier that is not included in the plurality of first identifiers, and when it is monitored that the data application side has received the lost data, the lost early warning cancellation information may be retransmitted to prompt that the lost problem has been repaired. The method for monitoring whether the data application side receives the lost data can also adopt the mode of comparing the first mark and the second mark.
FIG. 2 illustrates a data processing method according to one embodiment of the application, as illustrated in FIG. 2, the method comprising:
s202, acquiring target data, wherein the target data corresponds to a plurality of first identifiers. The data application party can receive target data sent by the data sender, specifically, the target data can comprise a plurality of pieces of data, each piece of data corresponds to one first identifier, and different pieces of data correspond to different first identifiers.
And S204, if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
And obtaining a deduplication instruction, wherein the deduplication instruction comprises repeated identifiers in the plurality of first identifiers. Specifically, whether repeated identifiers exist in a plurality of first identifiers corresponding to the target data can be detected, and if the repeated identifiers exist, a deduplication instruction is sent. When a deduplication instruction is received, a deduplication operation is performed, for example, deleting the corresponding target data or applying the corresponding target data only once.
FIG. 3 illustrates a data processing method according to one embodiment of the application, as illustrated in FIG. 3, the method comprising:
s302, target data are sent to a data application party, and the target data correspond to a plurality of second identifiers. The target data may include a plurality of pieces of data, each piece of data corresponding to one second identifier, and different pieces of data corresponding to different second identifiers. Preferably, the second identifier may be a uniquely coded identifier code.
And S304, if a retransmission instruction is acquired, retransmitting target data corresponding to a second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application side. Determining a missing identity of the plurality of first identities compared to the second identity by comparing the plurality of second identities with the plurality of first identities.
Fig. 4 is a schematic diagram of a data monitoring repair system according to an embodiment of the present application, where, as shown in fig. 4, a data acquisition system, a database and a data application system are sequentially connected to form an online real-time link system, and a data monitoring processing system is respectively connected to the data acquisition system, the database and the data application system.
Specifically, the data acquisition system generates unique serial codes ID at the same time when acquiring data or generating data, and the data and the ID are transmitted to the data application system on line. The data monitoring processing system acquires the ID sent by the data acquisition system through the database, acquires the ID received by the data application system from the online, and performs data restoration processing based on the ID and the ID. When the data repairing processing is carried out, the data monitoring processing system sends processing instructions and early warning information to the data acquisition system and the data application system.
Fig. 5 is a data processing method according to an embodiment of the present application. The method of fig. 5 may be performed by a data processing apparatus. As shown in fig. 5, the method includes:
s502, the data acquisition system acquires data and generates unique serialization IDs at the same time.
Specifically, the data acquisition system performs data acquisition, and the data enters the real-time link system in the form of sls/MetaQ/kafka/flume message queues. Each piece of data corresponds to a unique serialization ID which is generated when the data is acquired or generated, and the serialization ID comprises an acquisition system source, data generation time, a data number and a data generation mode.
S504, the data enter a database, and are subjected to ETL (Extract-Transform-Load) processing and summarized into a structured broad table.
S506, the database sends the data and the serialization ID to the data application system. The data normally enter the service system in the main link, and can be transmitted and applied in the main link by utilizing the self data quality protection mode of the original system.
S508, the data monitoring processing system acquires the serialization ID from the data application system. It should be noted that the serialized ID is an ID corresponding to data received by the data application system in a preset period, where the preset period may be determined according to a time of starting transmission of the data and a reasonable time consumption of a transmission process. The preset time period can be in the millisecond level or the second level, and can be specifically set according to the requirements of an application system.
S510, the database sends the serialization ID to the data monitoring processing system. Since only the serialized ID is transmitted, it is faster than the data monitoring processing system can acquire the serialized ID from the data application system. In particular, the data monitoring processing system may be a separate component, such as a Redis database, other in-memory database, or an alarm platform.
And S512, if the data monitoring processing system judges that the serialization ID acquired from the data application system is repeated, sending repeated early warning to the data application system.
And S514, the data application system deletes or does not process the corresponding online repeated data.
S516, after the data application system is successfully processed, the early warning is canceled.
And S518, if the data monitoring processing system judges that the serial ID obtained from the data application system lacks part of ID compared with the serial ID directly transmitted by the database, the loss early warning is transmitted to the data acquisition system.
S520, the data acquisition system resends the data corresponding to the missing ID. And the data acquisition system performs data positioning based on the lack of the ID, obtains corresponding data and sends the corresponding data to the data application system.
S522, after the data application system successfully receives the data corresponding to the missing ID, the early warning is canceled.
According to the data processing method, the data can be repaired before the data is applied in the real-time link through the data monitoring processing system, so that error data can be corrected without perception under the condition that a main data link is not affected; meanwhile, the system can provide real-time monitoring and early warning functions, is independent of an original system of a main link, can form a pluggable decoupling assembly, and is convenient for iterative updating.
An electronic device according to an embodiment of the present application will be described in detail below with reference to fig. 6. Referring to fig. 6, at the hardware level, the electronic device includes a processor, optionally including an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an industry standard architecture (Industry Standard Architecture, ISA) bus, a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus, or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 6, but not only one bus or type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, forming the data processing device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired;
and if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
Or performs the following operations:
acquiring target data, wherein the target data corresponds to a plurality of first identifiers;
and if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
Or performs the following operations:
sending target data to a data application party, wherein the target data corresponds to a plurality of second identifiers;
and if the retransmission instruction is acquired, retransmitting the target data corresponding to the second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application party.
The method performed by the data processing apparatus as disclosed in the embodiment of fig. 1 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be embodied directly in the execution of a hardware decoding processor, or in the execution of a combination of hardware and software modules in a decoding processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
Of course, other implementations, such as a logic device or a combination of hardware and software, are not excluded from the electronic device of the present application, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or a logic device.
The embodiments of the present application also provide a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform the method of the embodiment of fig. 1, and in particular to perform the method of:
after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired;
and if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party.
Or performing the following method:
acquiring target data, wherein the target data corresponds to a plurality of first identifiers;
and if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
Or performing the following method:
sending target data to a data application party, wherein the target data corresponds to a plurality of second identifiers;
and if the retransmission instruction is acquired, retransmitting the target data corresponding to the second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application party.
Fig. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present application. Referring to fig. 7, in one software implementation, a data processing apparatus 700 may include: an acquisition unit 71 and a processing unit 72, wherein,
an acquisition unit 71 that acquires a plurality of first identifications corresponding to target data after the target data is transmitted from a data sender to a data applicator;
and the processing unit 72 is configured to send a deduplication instruction to the data application party if there are duplicate identifiers in the plurality of first identifiers, where the deduplication instruction includes the duplicate identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the duplicate identifiers in the target data by the data application party.
According to the data processing device provided by the embodiment of the application, after target data is transmitted from the data sender to the data application party, a plurality of first identifiers corresponding to the target data are obtained, whether the on-line real-time data are repeated is determined by judging whether repeated identifiers exist in the plurality of first identifiers, and if the repetition exists, a de-duplication instruction is sent to the data application party, so that the data application party performs de-duplication processing on the data corresponding to the repeated identifiers in the target data. The data processing device of the embodiment of the application can monitor and repair real-time data, reduce the time consumption of online data processing, correct error data without perception and influence the normal operation of online business.
Alternatively, as an embodiment, the processing unit 72:
before target data is transmitted from the data sender to the data application party, a plurality of second identifiers corresponding to the target data are acquired;
and if at least one second identifier in the plurality of second identifiers is not included in the plurality of first identifiers, transmitting a retransmission instruction to the data transmitter, wherein the retransmission instruction comprises the second identifiers which are not included in the plurality of first identifiers, and the retransmission instruction is used for the data transmitter to carry out retransmission processing on the data corresponding to the second identifiers which are not included in the plurality of first identifiers in the target data.
Optionally, as an embodiment, the second identifier is generated based on at least one of the following information: data acquisition source information, data generation time information, data number information and data generation mode information.
Alternatively, as an embodiment, the processing unit 72:
acquiring the repetition times of the repeated mark;
and sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identification and the repeated times.
Alternatively, as an embodiment, the processing unit 72:
if repeated data exist in the target data, repeated early warning information is sent to the data application party and/or the data sender, and the repeated early warning information comprises the repeated identification;
and if the target data lose data, sending loss early warning information to the data application party and/or the data sender, wherein the loss early warning information comprises second identifiers which are not included in the plurality of first identifiers.
The data processing device according to the embodiment of the present application may also execute the method executed by the data processing device in fig. 1, and implement the functions of the data processing device in the embodiment shown in fig. 1, which are not described herein again.
Fig. 8 is a schematic diagram of a data processing apparatus according to an embodiment of the present application. Referring to fig. 8, in one software implementation, a data processing apparatus 800 may include: a data acquisition unit 81 and a deduplication execution unit 82, wherein,
a data acquisition unit 81 that acquires target data corresponding to a plurality of first identifications;
a deduplication execution unit 82 that, if a deduplication instruction is obtained, identifies a duplicate of the plurality of first identifications included in the deduplication instruction; the corresponding target data is deleted or applied only once.
The data processing device according to the embodiment of the present application may also execute the method executed by the data processing device in fig. 2, and implement the functions of the data processing device in the embodiment shown in fig. 2, which are not described herein again.
Fig. 9 is a schematic diagram of a data processing apparatus according to an embodiment of the present application. Referring to fig. 9, in one software implementation, a data processing apparatus 900 may include: a data acquisition unit 91 and a retransmission execution unit 92, wherein,
a data transmitting unit 91 that transmits target data to a data application, the target data corresponding to a plurality of second identifications;
and a retransmission execution unit 92 configured to, when a retransmission instruction is acquired, retransmit target data corresponding to a second identifier included in the retransmission instruction and not included in the plurality of first identifiers to the data application.
The data processing device according to the embodiment of the present application may also execute the method executed by the data processing device in fig. 3, and implement the functions of the data processing device in the embodiment shown in fig. 3, which are not described herein again.
In summary, the foregoing description is only of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
Claims (15)
1. A data processing method, comprising:
after target data is transmitted from a data sender to a data application, a plurality of first identifiers corresponding to the target data are acquired;
if repeated identifiers exist in the plurality of first identifiers, sending a deduplication instruction to the data application party, wherein the deduplication instruction comprises the repeated identifiers, and the deduplication instruction is used for performing deduplication processing on data corresponding to the repeated identifiers in the target data by the data application party;
before target data is transmitted from the data sender to the data application party, a plurality of second identifiers corresponding to the target data are acquired;
determining whether the target data after transmission is in error relative to the target data before transmission by comparing a second identifier corresponding to the target data before transmission to a data application and a first identifier corresponding to the target data after transmission to the data application, if at least one second identifier in the second identifiers is not included in the first identifiers, determining that the target data after transmission is in error, and sending a retransmission instruction to the data sender, wherein the retransmission instruction comprises the second identifiers not included in the first identifiers, and the retransmission instruction is used for the data sender to retransmit the data corresponding to the second identifiers not included in the first identifiers in the target data.
2. The method of claim 1, the second identification generated based on at least one of: data acquisition source information, data generation time information, data number information and data generation mode information.
3. The method of claim 1, further comprising:
if repeated data exist in the target data, repeated early warning information is sent to the data application party and/or the data sender, and the repeated early warning information comprises the repeated identification;
and if the target data lose data, sending loss early warning information to the data application party and/or the data sender, wherein the loss early warning information comprises second identifiers which are not included in the plurality of first identifiers.
4. A method according to claim 3, further comprising:
when the data application side successfully performs de-duplication processing, sending repeated early warning cancellation information to the data application side and/or the data sender, wherein the repeated early warning cancellation information comprises the repeated identification;
and after the retransmission processing of the data sender is successful, sending loss early warning cancellation information to the data application party and/or the data sender, wherein the loss early warning cancellation information comprises second identifiers which are not included in the plurality of first identifiers.
5. The method of claim 1, further comprising:
acquiring target data, wherein the target data corresponds to a plurality of first identifiers;
and if the deduplication instruction is obtained, deleting or only applying the target data corresponding to the repeated identifier in the plurality of first identifiers included in the deduplication instruction once.
6. The method of claim 1, further comprising:
and if the retransmission instruction is acquired, retransmitting the target data corresponding to the second identifier which is not included in the plurality of first identifiers and is included in the retransmission instruction to the data application party.
7. A data processing apparatus comprising:
an acquisition unit for acquiring a plurality of first identifications corresponding to target data after the target data is transmitted from a data sender to a data application;
the processing unit is used for sending a de-duplication instruction to the data application party if repeated identifiers exist in the plurality of first identifiers, wherein the de-duplication instruction comprises the repeated identifiers and is used for performing de-duplication processing on data corresponding to the repeated identifiers in the target data by the data application party;
the processing unit:
before target data is transmitted from the data sender to the data application party, a plurality of second identifiers corresponding to the target data are acquired;
determining whether the target data after transmission is in error relative to the target data before transmission by comparing a second identifier corresponding to the target data before transmission to a data application and a first identifier corresponding to the target data after transmission to the data application, if at least one second identifier in the second identifiers is not included in the first identifiers, determining that the target data after transmission is in error, and sending a retransmission instruction to the data sender, wherein the retransmission instruction comprises the second identifiers not included in the first identifiers, and the retransmission instruction is used for the data sender to retransmit the data corresponding to the second identifiers not included in the first identifiers in the target data.
8. The apparatus of claim 7, further comprising:
the data acquisition unit acquires target data, wherein the target data corresponds to a plurality of first identifiers;
a deduplication execution unit, if a deduplication instruction is obtained, repeating identifiers in the plurality of first identifiers included in the deduplication instruction; the corresponding target data is deleted or applied only once.
9. The apparatus of claim 7, further comprising:
and a retransmission execution unit configured to, if a retransmission instruction is acquired, retransmit target data corresponding to a second identifier included in the retransmission instruction and not included in the plurality of first identifiers to the data application side.
10. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, perform the method of claim 1 using the processor.
11. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, perform the method of claim 5 using the processor.
12. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, use the processor to perform the method of claim 6.
13. A computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of claim 1.
14. A computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of claim 5.
15. A computer readable medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310827031.3A CN116737710A (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310827031.3A CN116737710A (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
| CN201910728574.3A CN110597794B (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
Related Parent Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910728574.3A Division CN110597794B (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116737710A true CN116737710A (en) | 2023-09-12 |
Family
ID=68853700
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310827031.3A Pending CN116737710A (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
| CN201910728574.3A Active CN110597794B (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
Family Applications After (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910728574.3A Active CN110597794B (en) | 2019-08-08 | 2019-08-08 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (2) | CN116737710A (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111371900B (en) * | 2020-03-13 | 2022-07-12 | 北京奇艺世纪科技有限公司 | Method and system for monitoring health state of synchronous link |
| CN111913807B (en) * | 2020-08-13 | 2024-02-06 | 支付宝(杭州)信息技术有限公司 | Event processing method, system and device based on multiple storage areas |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107229660A (en) * | 2016-03-25 | 2017-10-03 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus of data deduplication |
| CN107608689B (en) * | 2017-08-07 | 2020-01-21 | 阿里巴巴集团控股有限公司 | Method and device for updating application program on client and electronic equipment |
| CN107656966A (en) * | 2017-08-28 | 2018-02-02 | 深圳市诚壹科技有限公司 | The method and server of a kind of processing data |
| CN107766508B (en) * | 2017-10-23 | 2021-06-15 | 深圳市中润四方信息技术有限公司 | Method and system for collecting and distributing data files |
| CN110099086B (en) * | 2018-01-31 | 2020-11-06 | 国广融合（北京）传媒科技发展有限公司 | Data transmission method based on fusion transmission system |
-
2019
- 2019-08-08 CN CN202310827031.3A patent/CN116737710A/en active Pending
- 2019-08-08 CN CN201910728574.3A patent/CN110597794B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110597794B (en) | 2023-07-25 |
| CN110597794A (en) | 2019-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109062809B (en) | Online test case generation method and device and electronic equipment | |
| CN107633016B (en) | Data processing method and device and electronic equipment | |
| CN110022259B (en) | Message arrival rate determining method and device, data statistics server and storage medium | |
| CN110597794B (en) | Data processing method and device and electronic equipment | |
| CN111309635B (en) | Test case generation method, device, server and storage medium | |
| CN110806971A (en) | Version testing method and device and electronic equipment | |
| CN111046310A (en) | Page processing method, device, server and computer readable storage medium | |
| CN112835885B (en) | Processing method, device and system for distributed form storage | |
| CN110781153B (en) | Cross-application information sharing method and system based on block chain | |
| CN111355765B (en) | Network request processing and sending method and device | |
| CN112969198A (en) | Data transmission method, terminal and storage medium | |
| CN111897826A (en) | Parameter information updating method and device, electronic equipment and readable storage medium | |
| CN116069751A (en) | Information processing method, apparatus, device and computer readable storage medium | |
| CN108965108B (en) | Message pushing method and related equipment | |
| CN111063182A (en) | Disaster monitoring method and device and electronic equipment | |
| CN108390770B (en) | Information generation method and device and server | |
| CN111833608B (en) | Road condition information processing method and device | |
| CN110740062B (en) | Breakpoint resume method and device | |
| CN109815083B (en) | Application crash monitoring method and device, electronic equipment and medium | |
| CN115203178A (en) | Data quality inspection method and device, electronic equipment and storage medium | |
| CN114416420A (en) | Equipment problem feedback method and system | |
| CN110362464B (en) | Software analysis method and equipment | |
| CN110866225A (en) | Risk control method and device, electronic equipment and storage medium | |
| CN112910936A (en) | Data processing method, device and system, electronic equipment and readable storage medium | |
| CN111371650B (en) | Flow forwarding-based quasi-production equipment testing method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |