CN116232551A - Data exchange method, transmitting end, exchange system, electronic equipment and medium - Google Patents

Data exchange method, transmitting end, exchange system, electronic equipment and medium Download PDF

Info

Publication number
CN116232551A
CN116232551A CN202211643752.0A CN202211643752A CN116232551A CN 116232551 A CN116232551 A CN 116232551A CN 202211643752 A CN202211643752 A CN 202211643752A CN 116232551 A CN116232551 A CN 116232551A
Authority
CN
China
Prior art keywords
data
target data
receiving
bit
state bit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211643752.0A
Other languages
Chinese (zh)
Inventor
张威
吴丽卿
陈毅
宾伟
邬传朋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Taiji Shuzhi Technology Co ltd
Original Assignee
Shenzhen Taiji Shuzhi Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Taiji Shuzhi Technology Co ltd filed Critical Shenzhen Taiji Shuzhi Technology Co ltd
Priority to CN202211643752.0A priority Critical patent/CN116232551A/en
Publication of CN116232551A publication Critical patent/CN116232551A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/602—Providing cryptographic facilities or services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1829—Arrangements specially adapted for the receiver end
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
    • H04L1/1867—Arrangements specially adapted for the transmitter end
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The embodiment of the application discloses a data exchange method, a sending end, an exchange system, electronic equipment and a medium, which can only need to retransmit partial abnormal data when the data exchange is abnormal, and does not need to retransmit all the data. The data exchange method comprises the following steps: reading a target data set from a pre-library, wherein the target data set comprises at least one target data, the attribute of the data to be processed in the pre-library comprises a state bit, the target data is the data to be processed, the state bit of which is a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit; placing target data into a transmission queue; and reading the target data from the transmission queue and transmitting the target data to the receiving end.

Description

Data exchange method, transmitting end, exchange system, electronic equipment and medium
Technical Field
The application belongs to the technical field of data exchange, and particularly relates to a data exchange method, a sending end, a data exchange system, electronic equipment and a computer readable storage medium.
Background
Data warehouse technology refers to technology that a large amount of original data is extracted, converted and loaded to a target warehouse, and is widely applied to the field of large data nowadays.
However, the existing data warehouse technology still has some problems, which are embodied in that when the data exchange is abnormal, all data needs to be retransmitted, and the repeated exchange of the data is caused.
Disclosure of Invention
The embodiment of the application provides a data exchange method, a transmitting end, a data exchange system, electronic equipment and a computer readable storage medium, which can solve the problem that when data exchange is abnormal, all data are required to be retransmitted, so that repeated data exchange is caused.
In a first aspect, an embodiment of the present application provides a data exchange method, where the method is applied to a transmitting end, and includes:
reading a target data set from a pre-library, wherein the target data set comprises at least one target data, the attribute of the data to be processed in the pre-library comprises a state bit, the target data is the data to be processed, the state bit of which is a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit;
placing target data into a transmission queue;
and reading the target data from the transmission queue and transmitting the target data to the receiving end.
According to the embodiment of the application, the state bit is set for each piece of data, so that the state of the corresponding data in the data exchange process can be recorded, the data in the abnormal state is checked, and the data in the abnormal state is retransmitted. Therefore, when the data exchange process is abnormal, only partial abnormal data is retransmitted, and all data are not retransmitted.
In a possible implementation manner of the first aspect, the attribute of the data to be processed further includes a lot identifier, where the lot identifier is used to identify a lot to which the data to be processed belongs, and the method further includes:
if receiving the receiving state information sent by the receiving end, updating the state bit of the target data with the batch identifier being the first batch identifier into a receiving state bit, wherein the receiving state information comprises the first batch identifier and the receiving state bit;
if the insertion state information sent by the receiving end is received, the state bit of the target data of which the batch identifier is the second batch identifier is updated to be the insertion state bit, and the insertion state information comprises the second batch identifier and the insertion state bit.
According to the method and the device for monitoring the target data, the target data of the corresponding batch are monitored through the receiving state information and the inserting state information sent by the receiving end, when the target data are abnormal at the receiving end, the target data can be timely retransmitted, and the situation that the target data cannot be processed when the target data are abnormal at the receiving end is avoided.
In a possible implementation manner of the first aspect, before reading the target data set from the pre-library, the method further includes:
extracting source data from a source database, and labeling batch identification for the source data;
Encrypting the source data belonging to the same batch identifier to obtain encrypted data;
compressing the encrypted data to obtain data to be processed;
and storing the data to be processed into a pre-library, and giving an initial state bit to the data to be processed.
According to the embodiment of the application, before the data are transmitted, the data are encrypted, so that the safety of the data in the transmission process is ensured. After encryption, the data is compressed, so that the number of data transmission strips is reduced, and the transmission efficiency is improved.
In a possible implementation manner of the first aspect, sending the target data to the receiving end includes:
and sending the target data to the receiving end through the encryption channel.
According to the embodiment of the application, the encryption channel is established between the sending end and the receiving end, and the sending end sends data to the receiving end through the encryption channel, so that the safety in the data transmission process is guaranteed.
In a possible implementation manner of the first aspect, before reading the target data set from the pre-library, the method further includes:
and updating the status bit of the target data with the status bit of the sending queue, the sending status bit, the receiving queue busy status bit and the inserting failure status bit into the abnormal status bit.
In a second aspect, an embodiment of the present application provides a transmitting end, including:
and a reading module: the method comprises the steps that a target data set is read from a pre-library, the target data set comprises at least one target data, the attribute of data to be processed in the pre-library comprises a state bit, the target data is the data to be processed, the state bit is a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit;
a queue module: for placing the target data into a transmit queue;
and a sending module: the method is used for reading the target data from the sending queue and sending the target data to the receiving end.
In a third aspect, an embodiment of the present application provides a data exchange system, including:
the transmitting end is used for reading a target data set from the pre-database, the target data set comprises at least one target data, the attribute of the data to be processed in the pre-database comprises a state bit, the target data is the data to be processed, the state bit is a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit; placing target data into a transmission queue; and reading the target data from the transmission queue and transmitting the target data to the receiving end.
The receiving end is used for receiving the target data and inserting the target data into the target database.
In a possible implementation manner of the third aspect, the method includes:
the receiving end is specifically configured to: receiving target data; if the receiving queue is full, generating receiving state information according to the queue condition, and sending the receiving state information to a sending end; if the receiving queue is not full and the status bit of the target data is an inserted status bit or a non-inserted status bit, generating receiving status information according to the status bit of the target data and sending the receiving status information to the sending end; if the state bit of the target data is the abnormal state bit of the receiving end, the target data is put into a sending queue, receiving state information is generated, the receiving state information is sent to the sending end, and the state bit of the target data is updated from the abnormal state bit of the receiving end to the state bit of the receiving queue; wherein the receiving status information includes a first lot identification and a receiving status bit;
the receiving end is specifically configured to: reading target data in the receiving queue; inserting the target data into a target database, generating insertion state information according to the insertion state, and sending the insertion state information to a sending end, wherein the insertion state information comprises a second batch identifier and an insertion state bit;
the transmitting end is also used for: if receiving the receiving state information sent by the receiving end, updating the state bit of the target data with the batch identification as the first batch identification into a receiving state bit;
And if the insertion state information sent by the receiving end is received, updating the state bit of the target data of which the batch identifier is the second batch identifier into the insertion state bit.
In a fourth aspect, an embodiment of the present application provides an electronic device, including:
a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the method according to any one of the above first aspects when executing the computer program.
In a fifth aspect, embodiments of the present application provide a computer-readable storage medium storing a computer program which, when executed by a processor, implements a method as in any one of the first aspects described above.
In a sixth aspect, embodiments of the present application provide a computer program product for, when run on a terminal device, causing the terminal device to perform the method of any one of the first aspects described above.
It will be appreciated that the advantages of the second to sixth aspects may be found in the relevant description of the first aspect, and are not described here again.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description will briefly introduce the drawings that are needed in the embodiments or the description of the prior art, it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is an interaction schematic diagram of a transmitting end and a receiving end provided in an embodiment of the present application;
fig. 2 is an interaction schematic diagram of another transmitting end and a receiving end according to an embodiment of the present application;
FIG. 3 is a flow chart of a data exchange method according to an embodiment of the present disclosure;
fig. 4 is a schematic block diagram of a structure of a transmitting end according to an embodiment of the present application;
fig. 5 is a schematic block diagram of a data exchange system according to an embodiment of the present application;
fig. 6 is a schematic block diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should also be understood that the term "and/or" as used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
In addition, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and are not to be construed as indicating or implying relative importance.
Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
Data warehouse technology (ETL) is used to describe the process of extracting, converting, and loading data from a source to a destination. The extraction is to abstract various original format data to form a unified data format, and the unified data format is firstly put into a buffer area and cannot directly enter a data warehouse to wait for the next conversion operation. The transformation may be data processing with some rules and methods according to business needs. The loading is to load the converted data into the target database, and the loading mode of the data generally has the following two important types: full load and incremental load, the full load is generally data load for the first time, the process is longer, and there are cases that the service data has historical full data to be updated continuously. The incremental loading is realized by extracting incremental data generated in the current period from a source system according to a certain period in a mode of customizing a planning task after data initialization, and the general incremental loading is realized by a trigger mode, a time stamp mode, a full table comparison mode and a database log mode.
However, as mentioned in the background, the existing data warehouse technology has some problems, which are specifically that when the data exchange is abnormal, all data needs to be retransmitted, and thus the data exchange is repeated.
Based on this, the embodiment of the application provides a data exchange method, which uses status bits to monitor the status of corresponding data in the data exchange process by setting the status bits to the data. Recording and retransmitting the data which is abnormal in the data exchange process. When the data exchange is abnormal, only partial abnormal data is retransmitted, and all data are not retransmitted.
Referring to fig. 1, fig. 1 is a schematic flowchart of a data exchange method provided in an embodiment of the present application, where the data exchange method provided in the embodiment of the present application may be applied to an electronic device such as a mobile phone, a tablet computer, a server, a wearable device, a vehicle-mounted device, an augmented reality (augmented reality, AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (personal digital assistant, PDA), etc., where the server includes but is not limited to an independent server or a cloud server. The embodiment of the application does not limit the specific type of the electronic device. As shown in fig. 1, the method comprises the steps of:
S101, the transmitting end reads the target data set from the front-end library.
Specifically, the sending end may extract a target data set from the pre-library when the sending end is started for the first time and/or a preset period passes, where the target data set includes at least one target data, and the target data is to-be-processed data with a state bit being a preset state bit.
The sending end can be a module in the data compression assembly, and the data compression assembly also can comprise a receiving end corresponding to the sending end, wherein the data compression assembly is a configured data processing assembly in ETL software. The pre-library may be a database, and the pre-library is used for storing data to be processed, which will be simply referred to as data for convenience of description. The state bit is an attribute of data and is used for identifying what state the data is in a data exchange process, the preset state bit comprises an initial state bit and an abnormal state bit, the preset state bit is used for representing data needing to be transmitted, and the initial state bit is used for representing data which is not transmitted and needs to be transmitted. The exception status bit is used to indicate data that has been transmitted, but that has been abnormal during transmission, and that needs to be transmitted again. The abnormal state bits may include a transmitting-end abnormal state bit indicating abnormal data in the transmitting end and a receiving-end abnormal state bit indicating abnormal data in the receiving end.
In some embodiments, the data may be identified using a lot identification, i.e., the attributes of each data also include a lot identification, which is used to identify the lot to which the data belongs.
By way of example, and not limitation, the following table sets forth a status bit of data and its meaning in the course of data exchange.
Figure BDA0004008896460000071
Figure BDA0004008896460000081
TABLE 1
As shown in the table, the status code is used to identify the status of the data processing, so that the status of the data can be monitored, and the transmitting end can retransmit the data when the data is abnormal.
For example, in some embodiments, the initial status bit is 1 and the exception status bits are comprised of-2 and-1, where-2 is the receiver exception status bit, i.e., data representing an exception in the receiver, and-1 is the sender exception status bit, i.e., data representing an exception in the sender. Extracting target data from the pre-library means extracting data with status bits of-2, -1, and 0 from the pre-library.
It should be noted that the status codes and the meanings thereof are merely exemplary of the present invention, and are not limiting, and other status codes and meanings thereof may exist according to practical situations.
In some embodiments, before the sender reads the target data set from the pre-library, the method further comprises extracting source data from the source database, and labeling the source data with a lot identification. Encrypting the source data belonging to the same batch identifier to obtain encrypted data. And compressing the encrypted data to obtain the data to be processed. And storing the data to be processed into a pre-library, and endowing the data to be processed with the initial state bit.
Specifically, the sender firstly acquires the read and write rights of the source database, and configures a table input step for guiding the sender to read which library tables of the source database. The sending end obtains source data from one or more library tables of the source database according to the table input step, marks the obtained source data with a batch identifier, and the batch identifier is used for identifying the batch to which the source data belongs.
After extracting the source data, the transmitting end encrypts the source data belonging to the same batch to obtain encrypted data. The encryption key may be a batch identifier of the source data, the encryption algorithm may be an advanced encryption standard (Advanced Encryption Standard, AES) or a data encryption standard (Data Encryption Standard, DES), and the embodiment of the present application does not limit the encryption method.
After the encrypted data is obtained, the encrypted data can be compressed to obtain the data to be processed. In particular, a specific separator may be used to splice a plurality of pieces of encrypted data belonging to the same lot identifier into one piece of data. And finally, storing the data to be processed into a pre-library, and giving an initial state bit to the data to be processed, wherein the initial state bit indicates that the batch of data is not transmitted and needs to be transmitted.
In some embodiments, before the sender extracts the target data from the pre-database, the sender may further perform a lock table clearing operation on the pre-database, and update the status bit of the data in the pre-database that has no lot identifier and the status bit is the status bit of the send queue to the initial status bit. Wherein the transmit queue status bit is used to indicate that data is in the transmit queue.
S102, the transmitting end puts the target data into a transmitting queue.
In some embodiments, the sender may update the status bit of the target data to the send queue status bit after placing the target data in the send queue.
S103, the sending end reads the target data from the sending queue and sends the target data to the receiving end.
Specifically, when the transmitting end monitors that data exists in the data transmission queue, the transmitting end reads the data from the data transmission queue and transmits the target data to the receiving end.
In some embodiments, the sender sends the target data to the receiver over an encrypted channel. The channel object of Netty can be used for constructing a transmission channel, sslhander can be configured for the channel object when the channel object is initialized, and interception and secure socket layer (Secure Socket Layer, SSL) encryption and decryption are performed when data enter and exit the transmission channel. Of course, there are other ways to implement the encryption channel, and the application is not limited to the encryption channel.
In some embodiments, the transmitting end updates the status bit of the target data with the lot identification as the first lot identification to the receiving status bit if the receiving end receives the receiving status information sent by the receiving end. If the transmitting end receives the insertion state information transmitted by the receiving end, the state bit of the target data of which the batch identifier is the second batch identifier is updated to be the insertion state bit.
The receiving status information comprises a first batch identifier and a receiving status bit, wherein the first batch identifier is a batch identifier of target data which is received by the receiving end, and the receiving status bit is used for indicating a result of the receiving end for receiving the target data. The insertion status information includes a second lot identifier, which is a lot identifier of the target data of the insertion operation performed by the receiving end, and an insertion status bit for indicating an insertion result of the target data into the target database by the receiving end.
In some embodiments, before the sender reads the target data with the status bit being the preset status bit from the pre-library, the method further includes: the transmitting end updates the status bit of the target data, which is the status bit of the transmitting queue, the status bit of the receiving queue busy and the status bit of the inserting failure status bit, into the abnormal status bit.
The sending queue status bit is used for indicating that target data enter a sending queue, the sending status bit is used for indicating that the target data are sent to a receiving end, the receiving queue busy status bit is used for indicating that the receiving queue of the receiving end is full, the target data cannot be received any more, the insertion failure status bit is used for indicating that the target data are abnormal in the process that the receiving end is inserted into a target database, the abnormal status bit is used for indicating that the target data are transmitted, but the abnormal status bit is generated in the process of transmission, and the transmission is needed again.
By way of example and not limitation, a transmit queue status bit is indicated by 1, a transmit status bit is indicated by 2, a receive queue busy status bit is indicated by 4, an insertion failure status bit is indicated by 8, and an abnormal status bit is indicated by-1. Before the transmitting end reads target data with state bits being preset state bits from the pre-arranged library, the state bits of the data with state bits being 1, 2, 4 or 8 in the pre-arranged library are changed into abnormal state bits-1.
In some embodiments, after the transmitting end transmits the target data to the receiving end, the transmitting end may update the status bit of the target data from the status bit of the transmission queue to the transmission status bit, that is, the transmission status bit represents that the transmitting end transmits the target data to the receiving end.
S104, the receiving end receives the target data and inserts the target data into a target database.
Fig. 2 is a schematic diagram of another data exchange method according to an embodiment of the present application, as shown in fig. 2, in some embodiments, step S104 includes:
s201, a receiving end receives target data, processes the target data and generates receiving state information.
As an example and not by way of limitation, fig. 3 shows a schematic diagram of processing target data at a receiving end according to an embodiment of the present application, as shown in fig. 3, step S301 includes:
s301, a receiving end receives target data.
S302, whether the receiving queue is full.
Specifically, after receiving the target data, the receiving end processes the data according to the state of the data receiving queue, and if the receiving queue is full, the step goes to step S308, and if the receiving queue is not full, the step goes to step S302.
S303, whether the status bit of the target data belongs to the inserted status bit or the non-inserted status bit.
Specifically, the receiving end identifies the status bit of the target data, if the status bit belongs to the inserted status bit or the non-inserted status bit, the step goes to step S307, otherwise, the step goes to step S304. Wherein the inserted status bit represents that the target data has been inserted into the target database, the inserted status bit may include: the receiving end inserts the target data into the target database, but does not send the insertion state information to the sending end; the receiving end inserts the target data into the target database and successfully transmits the insertion state information to the transmitting end; the receiving end inserts the target data into the target database, but fails to send the insert information to the transmitting end. It will be appreciated that each of the 3 inserted states may be identified by a different state bit, i.e., the inserted state bits may also include 3 state bits.
The non-inserted status bit represents that the target data is not inserted into the target database, and the non-inserted status bit can comprise that the receiving end sends the target data into a receiving queue; the receiving end takes the target data out of the receiving queue. It will be appreciated that each of the 2 non-inserted states may be identified by a different state bit, i.e., the non-inserted state bits may also include 2 state bits.
S304, whether the status bit of the target data belongs to the abnormal status bit of the receiving end or the status bit of the abnormal batch received by the sending end.
Specifically, if the status bit of the target data belongs to the receiving end abnormal status bit or the sending end receives the abnormal batch status bit, the process goes to step S306, otherwise, the process goes to step S305.
The process of updating the status bit of the data into the abnormal status bit of the receiving end is specifically as follows: after the receiving end is restarted and/or a certain preset period passes, the receiving end identifies the data in the state that the receiving end sends the data into the receiving queue, the data in the state that the receiving end takes the data out of the receiving queue, and the data with the state bit being the abnormal state bit of the receiving end, the receiving end acquires the batch information of the data in the three states and sends the batch information to the sending end, and then the receiving end takes the state of the data in the three states as the abnormal state bit of the updating bit receiving end.
Accordingly, after receiving the batch information of the data in the three states sent by the receiving end, the sending end can update the state bit of the corresponding batch to the abnormal state bit of the receiving end, and the subsequent sending end can retransmit the abnormal data of the batches. After the sending end updates the status bit of the corresponding batch to the abnormal status bit of the receiving end, the sending end can send corresponding modification information to the receiving end, and after the receiving end receives the modification information sent by the sending end, the receiving end updates the abnormal status bit of the receiving end to the status bit of the receiving end receiving the abnormal batch, namely, the batch representing that the sending end has received the abnormal data.
S305, putting the target data into a receiving queue, generating receiving state information, and inserting the receiving queue state bit into the state bit of the target data.
Specifically, the receiving end puts the target data into a receiving queue, and generates receiving state information after putting the target data into the receiving queue. The receiving status information comprises a first batch identifier and a receiving status bit, the receiving status bit is a status bit used for indicating that the receiving end has received the target data in the transmitting end, and the first batch identifier is a batch identifier of the target data which is received by the receiving end. Then inserts a receive queue status bit into the status bit of the target data.
The insertion of the received status bit into the status bit of the target data indicates that the status bit of the target data does not conform to the expected status bit or that the status bit is empty, and the status bit needs to be reinserted.
S306, putting the target data into a receiving queue, generating receiving state information, and updating the state bit of the target information into the state bit of the receiving queue.
Specifically, the receiving end puts the target data into a receiving queue, generates receiving state information after putting the target data into the receiving queue, and the receiving state information is used for feeding back to the sending end that the target data has been received, wherein the receiving state information comprises a first batch identifier and a receiving state bit, the first batch identifier is a batch identifier of the target data which is received by the receiving end, and the receiving state bit is a state bit in the sending end and used for indicating that the receiving end has received the target data. Then updating the state bit of the target data from the abnormal state bit of the receiving end or the abnormal batch state bit of the receiving end to the state bit of the receiving queue.
S307, receiving state information is generated according to the state bit of the target data.
Specifically, the receiving status information includes a first lot identifier and a receiving status bit, where the first lot identifier is a lot identifier of target data that is received by the receiving end. If the status bit of the target data is the inserted status bit, the receiving status bit is the status bit used for indicating that the target data is successfully inserted in the transmitting end. If the status bit of the target data is not inserted, the receiving status bit is a status bit in the transmitting end, which is used for indicating that the receiving end has received the target data.
S308, generating receiving state information according to the queue condition.
Specifically, the generated receiving status information includes a first lot identifier of the target data and a receiving status bit, where the first lot identifier is a lot identifier of the target data that is received by the receiving end, and the receiving status bit is a receiving queue busy status bit.
S202, the receiving end sends receiving state information to the sending end.
S203, the sending end updates the status bit of the target data with the lot identification as the first lot identification to the receiving status bit.
Specifically, after receiving the receiving status information, the transmitting end updates the status bit of the target data with the lot identifier being the first lot identifier to the receiving status.
S204, the receiving end inserts the target data into the target database and generates insertion state information.
Specifically, the receiving end extracts target data from the receiving queue and inserts the target data into the target database, and generates insertion state information according to the insertion state, wherein the insertion state information comprises an insertion state bit and a second batch identifier, and the second batch identifier is a batch identifier of the target data which is subjected to insertion processing by the receiving end. If the insertion is successful, the insertion status bit is a status bit in the transmitting end, which is used for indicating that the receiving end successfully inserts the target data into the target database. If the target data fails to be inserted, the insertion status bit is a status bit in the sending end, which is used for indicating that the receiving end fails to insert the target data into the target database.
In some embodiments, in order to ensure the integrity of the inserted data, after the receiving end extracts the target data from the receiving queue, the target data may be screened, the target data whose data size is within the preset data size range is screened, so as to obtain screened data, and the screened data is inserted into the target database. The preset data volume range can be determined according to actual service conditions.
In some embodiments, after the receiving end extracts the target data from the receive queue, the receiving end updates the state of the target data from the receive queue state bit to a state bit indicating that the target data was extracted from the receive queue.
S205, the receiving end sends the insertion state information to the sending end.
S206, the sending end updates the status bit of the target data with the lot identification being the second lot identification to the insertion status bit.
Specifically, after receiving the receiving status information, the transmitting end updates the status bit of the target data of the lot identifier as the second lot identifier to the receiving status bit.
In some embodiments, the sender includes a data source engine for performing steps S101 and S102 and a data transfer engine for performing step S103.
In some embodiments, the receiving end includes a data receiving engine and a data processing engine, the data receiving engine being configured to perform steps S201 and S202. The data processing engine is used to perform steps S204 and S205. The data receiving engine is also used for extracting target data in the receiving queue, adding the head and the tail to the batch identification of the target data, namely splicing specific characters at the forefront and the last of the batch identification, and then sending the target data to the data processing engine. The data processing engine is also used for receiving the target data sent by the data transmission engine, and carrying out data verification according to the head and the tail of the batch identifier added by the receiving engine, namely determining whether the data is the data sent by the data receiving engine, and determining whether a batch of data is transmitted completely, if not, the data is considered to be inserted into the failure processing.
In some embodiments, multiple sending ends and multiple receiving ends may be provided, where the multiple sending ends and the multiple receiving ends complete the data exchange task in a distributed processing manner. Specifically, the data exchange tasks may be distributed according to the load conditions of the transmitting end and the receiving end using open source software xxl-job.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic of each process, and should not limit the implementation process of the embodiment of the present application in any way.
Corresponding to the data exchange method set forth in the above method embodiment, fig. 4 shows a schematic diagram of a transmitting end device provided in the embodiment of the present application, and for convenience of explanation, only a portion relevant to the embodiment of the present application is shown. As shown in fig. 4, the apparatus includes:
the reading module 41 is configured to read a target data set from a pre-library, where the target data set includes at least one target data, an attribute of data to be processed in the pre-library includes a status bit, the target data is data to be processed whose status bit is a preset status bit, and the preset status bit includes an initial status bit and an abnormal status bit;
A queue module 42 for placing the target data into a transmit queue;
the sending module 43 is configured to read the target data from the sending queue and send the target data to the receiving end.
It should be noted that, because the content of information interaction and execution process between the above devices/units is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be referred to in the method embodiment section, and will not be described herein again.
Corresponding to the data exchange method set forth in the above method embodiment, fig. 5 shows a schematic structural diagram of the data exchange system provided in the embodiment of the present application, and as shown in fig. 5, the data exchange system 500 includes a sending end 511 and a receiving end 522, where the sending end 511 is configured to read a target data set from a pre-library, place the target data in a sending queue, read the target data from the sending queue, and send the target data to the receiving end 522. The receiving end 522 is configured to receive the target data and insert the target data into the target database.
It should be noted that, because the content of the information interaction and the execution process between the sending end and the receiving end is based on the same concept as the method embodiment of the present application, specific functions and technical effects thereof may be found in the method embodiment section, and details are not repeated here.
Fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 6, the electronic device 6 of this embodiment includes: at least one processor 60 (only one shown in fig. 6), a memory 61 and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, the processor 60 implementing the steps in any of the various data exchange method embodiments described above when executing the computer program 62.
The electronic device 6 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The electronic device may include, but is not limited to, a processor 60, a memory 61. It will be appreciated by those skilled in the art that fig. 6 is merely an example of the electronic device 6 and is not meant to be limiting as the electronic device 6, and may include more or fewer components than shown, or may combine certain components, or different components, such as may also include input-output devices, network access devices, etc.
The processor 60 may be a central processing unit (Central Processing Unit, CPU), the processor 60 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 61 may in some embodiments be an internal storage unit of the electronic device 6, such as a hard disk or a memory of the electronic device 6. The memory 61 may in other embodiments also be an external storage device of the electronic device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the electronic device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used for storing an operating system, application programs, boot loader (BootLoader), data, other programs, etc., such as program codes of the computer program. The memory 61 may also be used for temporarily storing data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Embodiments of the present application also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements steps that may implement the various method embodiments described above.
Embodiments of the present application provide a computer program product which, when run on a mobile terminal, causes the mobile terminal to perform steps that may be performed in the various method embodiments described above.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application implements all or part of the flow of the method of the above embodiments, and may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, where the computer program, when executed by a processor, may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing device/terminal apparatus, recording medium, computer Memory, read-Only Memory (ROM), random access Memory (RAM, random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media. Such as a U-disk, removable hard disk, magnetic or optical disk, etc. In some jurisdictions, computer readable media may not be electrical carrier signals and telecommunications signals in accordance with legislation and patent practice.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. For example, the apparatus/network device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The above embodiments are only for illustrating the technical solution of the present application, and are not limiting; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (10)

1. A data exchange method, applied to a transmitting end, the method comprising:
reading a target data set from a pre-library, wherein the target data set comprises at least one target data, the attribute of the data to be processed in the pre-library comprises a state bit, the target data is the data to be processed, the state bit of which is a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit;
Placing the target data into a transmission queue;
and reading the target data from the sending queue and sending the target data to a receiving end.
2. The data exchange method according to claim 1, wherein the attribute of the data to be processed further includes a lot identification for identifying a lot to which the data to be processed belongs, the method further comprising:
if receiving the receiving state information sent by the receiving end, updating the state bit of the target data of which the batch identifier is a first batch identifier into a receiving state bit, wherein the receiving state information comprises the first batch identifier and the receiving state bit;
and if the insertion state information sent by the receiving end is received, updating the state bit of the target data of which the batch identifier is the second batch identifier into an insertion state bit, wherein the insertion state information comprises the second batch identifier and the insertion state bit.
3. The data exchange method according to claim 2, wherein before reading the target data set from the pre-library, further comprising:
extracting source data from a source database, and labeling the source data with the batch identification;
Encrypting the source data belonging to the same batch identifier to obtain encrypted data;
compressing the encrypted data to obtain the data to be processed;
and storing the data to be processed into a pre-library, and endowing the data to be processed with the initial status bit.
4. The method of exchanging data according to claim 1, wherein said transmitting the target data to the receiving end includes:
and sending the target data to the receiving end through an encryption channel.
5. The data exchange method according to any one of claims 1 to 4, further comprising, before the target data set is read from the pre-library:
and updating the status bit which is the sending queue status bit, the sending status bit, the receiving queue busy status bit and the status bit of the target data inserted with the failure status bit into the abnormal status bit.
6. A transmitting terminal, comprising:
and a reading module: the method comprises the steps of reading a target data set from a pre-library, wherein the target data set comprises at least one target data, the attribute of data to be processed in the pre-library comprises a state bit, the target data is the data to be processed, the state bit of which is a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit;
A queue module: for placing the target data into a transmit queue;
and a sending module: and the target data is read from the sending queue and sent to a receiving end.
7. A data exchange system, comprising a transmitting end and a receiving end
The sending end is used for reading a target data set from a pre-database, the target data set comprises at least one target data, the attribute of the data to be processed in the pre-database comprises a state bit, the target data is the data to be processed with the state bit being a preset state bit, and the preset state bit comprises an initial state bit and an abnormal state bit; placing the target data into a transmission queue; reading the target data from the transmission queue and transmitting the target data to a receiving end;
the receiving end is used for receiving the target data and inserting the target data into a target database.
8. The data exchange system according to claim 7, wherein:
the receiving end is specifically configured to: receiving the target data; if the receiving queue is full, generating receiving state information according to the queue condition, and sending the receiving state information to the sending end; if the receiving queue is not full and the state bit of the target data is an inserted state bit or an uninserted state bit, generating the receiving state information according to the state bit of the target data and sending the receiving state information to the sending end; if the state bit of the target data is the abnormal state bit of the receiving end, the target data is put into a sending queue, receiving state information is generated, the receiving state information is sent to the sending end, and the state bit of the target data is updated from the abnormal state bit of the receiving end to the state bit of the receiving queue; wherein the receiving status information includes a first lot identification and a receiving status bit;
The receiving end is specifically configured to: reading the target data in the receiving queue; inserting the target data into a target database, generating insertion state information according to the insertion state, and sending the insertion state information to the sending end, wherein the insertion state information comprises the second batch identifier and an insertion state bit;
the transmitting end is further configured to: if the receiving status information sent by the receiving end is received, updating the status bit of the target data of which the batch identifier is the first batch identifier into the receiving status bit;
and if the insertion state information sent by the receiving end is received, updating the state bit of the target data of which the batch identifier is the second batch identifier into the insertion state bit.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 5 when executing the computer program.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 1 to 5.
CN202211643752.0A 2022-12-20 2022-12-20 Data exchange method, transmitting end, exchange system, electronic equipment and medium Pending CN116232551A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211643752.0A CN116232551A (en) 2022-12-20 2022-12-20 Data exchange method, transmitting end, exchange system, electronic equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211643752.0A CN116232551A (en) 2022-12-20 2022-12-20 Data exchange method, transmitting end, exchange system, electronic equipment and medium

Publications (1)

Publication Number Publication Date
CN116232551A true CN116232551A (en) 2023-06-06

Family

ID=86568742

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211643752.0A Pending CN116232551A (en) 2022-12-20 2022-12-20 Data exchange method, transmitting end, exchange system, electronic equipment and medium

Country Status (1)

Country Link
CN (1) CN116232551A (en)

Similar Documents

Publication Publication Date Title
CN110445860B (en) Message sending method, device, terminal equipment and storage medium
CN101867417B (en) Unidirectional transmission method based on optical fiber multi-way coupling
US20150373122A1 (en) Data processing method and apparatus
CN105516246A (en) Method for preventing application program downloading hijacking and server
CN114500690A (en) Interface data processing method and device, electronic equipment and storage medium
CN112346759A (en) Firmware upgrading method and device and computer readable storage medium
CN117640256B (en) Data encryption method, recommendation device and storage medium of wireless network card
CN108881124B (en) Method, system, storage medium and device for realizing high-performance communication between modules
CN116232551A (en) Data exchange method, transmitting end, exchange system, electronic equipment and medium
CN113849859A (en) Linux kernel modification method, terminal device and storage medium
CN116185512A (en) Drive loading method, device, equipment and medium for PTC driver
CN113676290A (en) Data transmission method based on fund system and related equipment
CN111859351A (en) Method, system, server and storage medium for writing information into chip
CN114490451A (en) Data encryption and data decryption method and device, storage medium and electronic device
EP3588841A1 (en) Method and device for executing an authentication scheme
CN111459899A (en) Log sharing method and device and terminal equipment
CN112269583B (en) Method for processing equipment operation abnormal file upgrade, server and storage medium
CN113129146B (en) Intelligent contract transaction processing method, equipment and storage medium
KR102592874B1 (en) Unmanned Aerial Vehicle, Apparatus for Generating Source Files Providing Confidential Information Protection of Unmanned Aerial Vehicle
CN110084003B (en) Method and system for accessing encryption lock based on middleware
CN115878351B (en) Message transmission method and device, storage medium and electronic device
CN113873035A (en) File transmission method and device, file transmission equipment and user terminal
CN116455892A (en) File transmission method, file transmission device and terminal equipment
CN115994180A (en) Data interface for data exchange between multi-service systems
CN111367582A (en) High-performance file type identification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination