CN112134951A

CN112134951A - Data transmission method and device, electronic equipment and storage medium

Info

Publication number: CN112134951A
Application number: CN202011003844.3A
Authority: CN
Inventors: 曹如海; 李强; 张炜; 宣月波
Original assignee: Ruishi Suzhou Video Technology Co Ltd; Suzhou Keda Technology Co Ltd
Current assignee: Ruishi Suzhou Video Technology Co Ltd; Suzhou Keda Technology Co Ltd
Priority date: 2020-09-22
Filing date: 2020-09-22
Publication date: 2020-12-25
Also published as: WO2022062347A1

Abstract

The invention relates to the technical field of data transmission, in particular to a data transmission method, a data transmission device, electronic equipment and a storage medium, wherein the method comprises the steps of providing a locally opened resource type, wherein the resource type corresponds to a resource source; receiving a resource registration request for at least one target resource type, wherein the resource registration request corresponds to a resource receiving port; determining a target resource source corresponding to the target resource type based on the resource registration request; extracting real-time data from a target resource source; and pushing the real-time data to at least one resource receiving port. The resource types are used for sharing the same real-time pushing mechanism, for the outside, the resource types are only known, specific resource sources are not known, the condition that a plurality of data interfaces are provided for the outside is avoided, the phenomenon that the pressure of the server is increased due to excessive external requests is avoided by reversely reversing and pushing, and the real-time performance of data transmission is improved.

Description

Data transmission method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of data transmission technologies, and in particular, to a data transmission method and apparatus, an electronic device, and a storage medium.

Background

An open platform refers to a software system that makes it possible for an external program to increase the functions of the software system or use the resources of the software system by disclosing its application programming interfaces or functions without changing the source code of the software system. When an external program wants to acquire real-time changing data inside a software system, the existing data transmission method is that the software system provides corresponding interface open data through an open platform, and the external system acquires the data regularly through an open data interface.

However, in the above data transmission method, if the number of external systems is increased and the real-time requirement is high, the external systems will request the interface at regular time continuously, which increases the pressure of the internal server, thereby reducing the real-time performance and affecting the efficiency of real-time data transmission.

Disclosure of Invention

In view of this, embodiments of the present invention provide a data transmission method, an apparatus, an electronic device, and a storage medium, so as to solve the problem of low transmission efficiency of real-time data.

According to a first aspect, an embodiment of the present invention provides a data transmission method, including:

providing a locally opened resource type, wherein the resource type corresponds to a resource source;

receiving a resource registration request for at least one target resource type, wherein the resource registration request corresponds to a resource receiving port;

determining a target resource source corresponding to the target resource type based on the resource registration request;

extracting real-time data from the target resource source;

and pushing the real-time data to at least one resource receiving port.

The data transmission method provided by the embodiment of the invention determines the target resource source corresponding to the target resource type by utilizing the corresponding relation between the resource type and the resource source, namely, the real-time data can be extracted from the target resource source and pushed to the resource receiving port, thereby achieving the purpose of pushing the data in real time.

With reference to the first aspect, in a first implementation manner of the first aspect, the pushing the real-time data to at least one resource receiving port includes:

determining status information of the at least one resource receiving port based on the resource registration request, the status information including one of normal, retry, exception, and recovery;

and pushing the real-time data to at least one resource receiving port according to the state information of the at least one resource receiving port.

According to the data transmission method provided by the embodiment of the invention, the real-time data is pushed based on the state information of at least one resource receiving port, so that the reliability of pushing the real-time data can be ensured.

With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the pushing the real-time data to at least one resource receiving port according to the state information of the at least one resource receiving port includes:

collecting the resource receiving port with the state information of the resource receiving port in a normal state;

pushing the real-time data to a resource receiving port in a normal state in parallel by using at least one thread;

judging whether the pushing of the preset thread is abnormal or not;

when the pushing of the preset thread is abnormal, stopping the real-time data pushing of the preset thread and setting the state information of the resource receiving port corresponding to the preset thread as retry;

performing data push corresponding to a retry mode based on the resource receiving port in the retry state;

or the like, or, alternatively,

the pushing the real-time data to at least one resource receiving port according to the state information of at least one resource receiving port includes:

collecting the resource receiving port with the state information of the resource receiving port as a retry state;

stopping pushing real-time data to the resource receiving port in the retry state and setting the state information of the resource receiving port as retry;

and the resource receiving port based on the retry state performs data pushing corresponding to the retry mode.

According to the data transmission method provided by the embodiment of the invention, the data are pushed to the resource receiving port in the normal state in parallel, so that the data transmission efficiency can be improved; and when the preset thread pushing is abnormal, setting the state information of the resource receiving port as retry, and pushing data in a retry mode to the resource receiving port, so that the normal state flow to the retry is realized to ensure the normal pushing of the data.

With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the performing, by the resource receiving port based on the retry state, data pushing corresponding to a retry mode includes:

another thread pushes the current push failure data to the resource receiving port in the retry state again;

judging whether the current push failure data is successfully pushed to the resource receiving port in the retry state within a preset condition;

when the current push failure data is successfully pushed to the resource receiving port in the retry state within a preset condition, setting the state information of the resource receiving port in the retry state to be in recovery;

performing data pushing corresponding to a recovery mode based on the recovery resource receiving port;

or the like, or, alternatively,

when the current push failure data is not successfully pushed to the resource receiving port in the retry state within a preset condition, setting the state of the resource receiving port in the retry state as abnormal;

and carrying out data push corresponding to the abnormal mode based on the resource receiving port in the abnormal state.

According to the data transmission method provided by the embodiment of the invention, if the data push is successful in the retry mode, the state information of the resource receiving port is set to be in recovery, so that the state flow from retry to recovery is realized, and a guarantee is provided for the data push; or when the resource receiving port in the retry state fails to successfully push the current push failed data within the preset conditions, the resource receiving port realizes the circulation from the retry state to the abnormal state, and a guarantee is provided for the normal push of the data.

With reference to the first implementation manner of the first aspect, in a fourth implementation manner of the first aspect, the pushing the real-time data to at least one resource receiving port according to the state information of the at least one resource receiving port includes:

collecting the state information of the resource receiving port as the resource receiving port in recovery;

and carrying out data pushing corresponding to the recovery mode based on the recovery resource receiving port.

With reference to the third implementation manner of the first aspect or the fourth implementation manner, in a fifth implementation manner of the first aspect, the performing, based on the resource receiving port in the recovery, data pushing corresponding to a recovery mode further includes:

extracting the configuration information of the resource receiving port in the recovery;

judging whether the interrupted data needs to be pushed or not based on the configuration information;

reading historical data from the interrupt position to the current position when the interrupted data needs to be pushed;

pushing the historical data to the resource receiving port in the recovery, and judging whether pushing failure occurs in the pushing process;

when the pushing failure does not occur, setting the state information of the resource receiving port in the recovery to be normal;

and carrying out data push corresponding to the normal mode based on the resource receiving port in the normal state.

According to the data transmission method provided by the embodiment of the invention, interruption continuous transmission can be carried out from the interruption position in the recovery mode, so that the data is ensured not to be lost; and when the historical data pushing fails, the state of the resource receiving port is transferred from the recovery to the normal state, and the guarantee is provided for recovering the interrupted pushing data.

With reference to the fifth implementation manner of the first aspect, in a sixth implementation manner of the first aspect, the performing, based on the resource receiving port in recovery, data pushing corresponding to a mode in recovery includes:

and when the pushing fails, setting the state information of the resource receiving port in the recovery as abnormal, and pushing data corresponding to the abnormal mode based on the resource receiving port in the abnormal state.

In the data transmission method provided by the embodiment of the invention, in the recovery mode, if the data push fails, the state of the resource receiving port is directly set to be abnormal, and the retry is not performed, so that the repeated retry of data transmission is avoided, and the real-time performance of data transmission is improved.

With reference to the first implementation manner of the first aspect, in a seventh implementation manner of the first aspect, the pushing the real-time data to at least one resource receiving port according to the state information of the at least one resource receiving port includes:

collecting the resource receiving port with the state information of the resource receiving port in an abnormal state;

With reference to the third embodiment, the sixth embodiment, or the seventh embodiment of the first aspect, in an eighth embodiment of the first aspect, the pushing data corresponding to the abnormal mode by the resource receiving port based on the abnormal state includes:

receiving the information that the state of the resource receiving port in the abnormal state is updated to be normal;

judging whether the set state of the resource receiving port in the abnormal state is abnormal or not;

when the set state of the resource receiving port in the abnormal state is abnormal, setting the state information of the resource receiving port in the abnormal state as recovery;

and carrying out data push corresponding to the recovery mode based on the recovery resource receiving port.

According to the data transmission method provided by the embodiment of the invention, when the state of the resource receiving port in the abnormal state is updated to be normal externally, the state of the resource receiving port is not directly set to be normal locally, but is set to be recovered firstly, and the abnormal state is transferred to the recovered state, so that the guarantee is provided for the normal transmission of the data.

With reference to the first implementation manner of the first aspect, in a ninth implementation manner of the first aspect, the pushing the real-time data to at least one resource receiving port according to the state information of the at least one resource receiving port, where the resource registration request carries configuration information, where the configuration information includes a data filtering condition, and the pushing includes:

filtering the real-time data by using the data filtering condition;

and pushing the filtered real-time data to at least one resource receiving port based on the state information of at least one resource receiving port.

According to the data transmission method provided by the embodiment of the invention, the real-time data is filtered before being transmitted, so that the transmitted real-time data can meet the external requirements.

According to a second aspect, an embodiment of the present invention further provides a data transmission apparatus, including:

the system comprises a providing module, a resource source module and a resource management module, wherein the providing module is used for providing a locally opened resource type, and the resource type corresponds to the resource source;

a receiving module, configured to receive a resource registration request for at least one target resource type, where the resource registration request carries configuration information of a corresponding resource receiving port;

a determining module, configured to determine, based on the resource registration request, a target resource source corresponding to the target resource type;

the extraction module is used for extracting real-time data from the target resource source;

and the pushing module is used for pushing the real-time data to at least one resource receiving port.

The data transmission device provided by the embodiment of the invention determines the target resource source corresponding to the target resource type by utilizing the corresponding relation between the resource type and the resource source, namely, the real-time data can be extracted from the target resource source and pushed to the resource receiving port, so that the purpose of pushing the data in real time is achieved, the resource type is utilized to share the same real-time pushing mechanism, so that a plurality of data interfaces are prevented from being provided for the outside, the pressure increase of a server caused by too many external requests is avoided by reversely reversing and pushing, and the real-time performance of data transmission is improved.

According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing therein computer instructions, and the processor executing the computer instructions to perform the data transmission method according to the first aspect or any one of the embodiments of the first aspect.

According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions for causing a computer to execute the data transmission method described in the first aspect or any one implementation manner of the first aspect.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 shows an optional application scenario of the data transmission method according to the embodiment of the present invention;

FIG. 2 is a flow chart of a method of data transmission according to an embodiment of the present invention;

FIG. 3 is a flow chart of a method of data transmission according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of an internal data flow mechanism according to an embodiment of the present invention;

FIG. 5 is a flow chart of a method of data transmission according to an embodiment of the present invention;

fig. 6 is an architecture diagram of a data transmission method according to an embodiment of the present invention;

fig. 7 is a block diagram of a structure of a data transmission apparatus according to an embodiment of the present invention;

fig. 8 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is an optional application scenario of the data transmission method according to the embodiment of the present invention. As shown in fig. 1, the data transmission system includes an electronic device (i.e., may be referred to as local hereinafter) and at least one external system (may be referred to as external program).

The electronic device establishes a corresponding relation between the resource type and the resource source in the electronic device, provides the resource type to the outside, and the external device selects a target resource type required by the external device from the resource types provided by the electronic device and sends a resource registration request to the electronic device.

After receiving a resource registration request sent by an external system, the electronic device determines a target resource source corresponding to the target resource type by using the corresponding relationship between the resource type and the resource source, extracts real-time data from the target resource source, and pushes the real-time data according to the state information of the resource receiving port of each external system.

For an external system, the external system does not need to request for acquiring data at regular time, and only needs to provide a resource receiving port, so that corresponding data can be received when the data exists, only the service processing needs to be concentrated, the corresponding butt joint work does not need to be repeatedly processed, and the function expansion is easy.

In the data transmission system, different types of open data sources are abstracted into resource types in the electronic equipment, and an external system only knows the resource types but not the specific internal data sources, so that the internal data sources are easy to switch, an external open interface is not required to be provided in the electronic equipment, the external system cannot sense the change of the data sources, and the internal migration and expansion are convenient. The details of the data transmission method will be described in detail below.

In accordance with an embodiment of the present invention, there is provided a data transmission method embodiment, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.

In this embodiment, a data transmission method is provided, which can be used in the above-mentioned electronic devices, such as a server, a computer, a tablet computer, and the like, and fig. 2 is a flowchart of the data transmission method according to the embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:

and S11, providing the locally opened resource types.

Wherein the resource type corresponds to a resource source.

The resource type is a number description of a local message to be opened to the outside, wherein a message subscription push system is maintained locally. The format of the specific number of the resource type can be correspondingly set according to the actual situation, and the external world can know the corresponding resource through the resource type only by ensuring. For example, it may be composed of a resource type name, a resource type description, and an outbound format of a resource message. The electronic equipment provides the external system with the knowledge, and the external system can know the data use of the external representation of the resource type according to the resource type, so that the electronic equipment can send a resource registration request to the electronic equipment in the subsequent step to request the real-time data corresponding to the resource type which the electronic equipment wants to acquire.

When the electronic device acquires a newly added resource source, the corresponding relationship between the resource type and the resource source is updated only in the electronic device based on the newly added resource source and the corresponding resource type thereof, and no external interface needs to be additionally arranged. The resource source may be a third-party data system, a database, a data engine or a data warehouse, etc., and the specific form of the resource source is not limited, and only the resource source can provide data.

S12, a resource registration request for at least one target resource type is received.

The resource registration request corresponds to a resource receiving port.

The electronic device provides the local resource type for the external system to know, and then the external system can request to acquire the real-time data corresponding to the target resource type by sending a resource registration request to the electronic device based on the self requirement. And each resource registration request corresponds to the information of a resource receiving port, and the resource receiving port is used for receiving the real-time data corresponding to the target resource type.

As shown in fig. 1, the resource registration request received by the electronic device may be a resource registration request for the same target resource type sent by at least one external system, and then the electronic device may perform the same processing on the same target resource type and send the same processing to different external systems respectively.

For example, the external system 1 corresponds to the resource receiving port a, the external system 2 corresponds to the resource receiving port b, and the external system 3 corresponds to the resource receiving port c. If the electronic device receives the resource registration request for the same target resource type sent by the external system 1, the external system 2, and the external system 3, the electronic device may determine the resource receiving ports corresponding to the external systems and the target resource receiving ports corresponding to the target resource types by using the resource registration request. Subsequently, after the real-time data are extracted, the real-time data can be respectively pushed to the corresponding resource receiving ports, so that the real-time data can be received by each external system.

Optionally, the resource registration request received by the electronic device may also be a resource registration request for at least one target resource type sent by at least one external system, and the electronic device may perform the same processing on the resource registration requests of the same target resource type and then send the resource registration requests to the corresponding resource receiving ports respectively.

For example, continuing with the embodiment described above, external system 1 requests target resource type c2, external system 2 requests target resource type c1, and external system 3 requests target resource type c 2. Since the external system 1 and the external system 3 request the same target resource type, the same operation can be performed on the external system 1 and the external system 3 in the electronic device, and after the real-time data 1 is subsequently extracted, the real-time data is respectively pushed to the corresponding resource receiving port a and the corresponding resource receiving port c; the electronic device may extract the real-time data 2 for the external system 2, and push the real-time data 2 to the corresponding resource receiving port b.

The resource registration request received by the electronic device is not limited at all, and may be set according to actual conditions. In the following description, at least one resource registration request for the same target resource type received by the electronic device is described in detail as an example.

S13, based on the resource registration request, determining the target resource source corresponding to the target resource type.

As described above, after receiving the resource registration request sent by each external system, the electronic device may determine the target resource source corresponding to the target resource type by querying the corresponding relationship between the target resource type and the resource source.

For example, an operator and at least one main listener are maintained in the electronic device, and after the electronic device receives the resource registration request, the operator determines the main listener to be started by using a target resource type corresponding to the resource registration request. After the target resource type is determined, starting a main monitor corresponding to the target resource type to extract real-time data from a target resource source corresponding to the target resource type. When at least one external system registers the corresponding resource type, the master monitor is triggered to start so as to avoid wasting resources when the monitors are in idle operation.

The electronic device may be provided with a general main listener, where the interfaces provided by the main listener include a monitoring source interface and a real-time data push interface, where the monitoring source interface is configured to extract real-time data from a resource source, and the real-time data push interface is configured to push the extracted real-time data to a corresponding resource receiving port. When the resource sources are of multiple types, adapters corresponding to the resource sources of the respective types can be further arranged in the universal master listener, and are used for performing corresponding data conversion and the like.

Of course, it is also possible to create master listeners in the electronic device corresponding to resource sources, one for each resource source. For example, where the resource sources include a database, a data warehouse, and a digital engine, then a corresponding master listener may be created for the database, the data warehouse, and the digital engine, respectively. The specific setting mode of the main monitor is not limited at all, and corresponding setting can be performed according to actual conditions, and only the main monitor needs to be ensured to extract real-time data from the determined resource source and push the extracted real-time data to a corresponding resource receiving port.

Specifically, as described above, a message subscription pushing system is maintained in the electronic device, after the electronic device receives a resource registration request, a newly added resource source is added to the message subscription pushing system along with a difference in resource source, and the message subscription pushing system can be updated only by updating a corresponding relationship between a resource type and a resource source, so that the resource type in the electronic device can be known by an external system in time.

And S14, extracting the real-time data from the target resource source.

As described above, the electronic device may extract real-time data from the target resource source using the master listener determined by the target resource source.

And S15, pushing the real-time data to at least one resource receiving port.

The external system knows the data use of the external representation of the resource type according to the description of the resource type, registers the resource type corresponding to the data which the external system wants to acquire in an interface provided in an external open platform through an internal program, and provides a related resource receiving port. After the electronic equipment extracts the real-time data, the electronic equipment pushes the real-time data to the resource receiving ports by using the state information of the resource receiving ports.

Optionally, the resource registration request may also carry configuration information such as a filtering rule for the data, a frequency of data push, whether the data needs to be continuously transmitted at a breakpoint, and the like. The configuration information is stored by the internal message subscription pushing system, then the internal resource type is registered by the external system, the data is acquired according to the processing method of the resource type, and then the configuration information of the external system is utilized to carry out regular filtering processing to determine whether to push the data to the external system. And when the fact that the data needs to be pushed to the external system is determined, the real-time data is pushed to the external system by utilizing the state information of each resource receiving port.

The data transmission method provided by this embodiment determines the target resource source corresponding to the target resource type by using the corresponding relationship between the resource type and the resource source, that is, the real-time data can be extracted from the target resource source and pushed to the resource receiving port, so as to achieve the purpose of pushing the data in real time.

In this embodiment, a data transmission method is provided, which can be used in the above-mentioned electronic devices, such as a server, a computer, a tablet computer, and the like, and fig. 3 is a flowchart of the data transmission method according to the embodiment of the present invention, as shown in fig. 3, the flowchart includes the following steps:

and S21, providing the locally opened resource types.

Wherein the resource type corresponds to a resource source.

Please refer to S11 in fig. 2 for details, which are not described herein.

S22, a resource registration request for at least one target resource type is received.

The resource registration request corresponds to a resource receiving port.

Please refer to S12 in fig. 2 for details, which are not described herein.

S23, based on the resource registration request, determining the target resource source corresponding to the target resource type.

Please refer to S13 in fig. 2 for details, which are not described herein.

And S24, extracting the real-time data from the target resource source.

Please refer to S14 in fig. 2 for details, which are not described herein.

And S25, pushing the real-time data to at least one resource receiving port.

The electronic equipment also determines the state information of at least one resource receiving port based on the resource registration request, and pushes real-time data to at least one resource receiving port according to the state information of at least one resource receiving port. Wherein the status information includes one of normal, retry, exception, and recovery.

As described above, the electronic device pushes real-time data to each resource receiving port according to the status information of the resource receiving port. Specifically, the following processing modes can be classified:

(1) resource receiving port in normal state

The electronic equipment collects the resource receiving port of which the state information is in a normal state, and pushes data corresponding to a normal mode based on the normal resource receiving port. For the data push in the normal mode, see the following description of S252 and its subsequent steps.

(2) Resource receiving port in retry state

The electronic device collects the resource receiving port whose status information is the retry status, and performs data push corresponding to the retry mode based on the retry resource receiving port. The data push in the retry mode can be referred to the related description of S256 below.

(3) Resource receiving port in abnormal state

The electronic equipment collects the resource receiving port of which the state information is in an abnormal state, and pushes data corresponding to the abnormal mode based on the abnormal resource receiving port. Wherein, the data push of the abnormal mode can be referred to the related description of step (6) in S255 below.

(4) Resource receiving port capable of recovering state

The electronic equipment collects the state information of the resource receiving port as the resource receiving port in the recovery state, and performs data push corresponding to the recovery mode based on the resource receiving port in the recovery state. Wherein, the data push of the recovery mode can be referred to the related description of step (4) in S256 below.

As can be seen from the above, the electronic device performs data push in different modes based on the state of each resource receiving port. It should be noted that, in the data pushing process, the state information of the same resource receiving port may change. For example, it may change from a normal state to a retry state, from a retry state to an abnormal state, from an abnormal state to a recovering state, and so on. In the following description, the state change after the resource reception and the processing mode corresponding to each state are described in detail, taking the state information of the resource reception port as a normal state as an example.

Specifically, the step S25 includes the following steps:

s251, determining status information of at least one resource receiving port based on the resource registration request.

Since the resource registration request corresponds to the resource receiving interface, and the real-time status information of each resource receiving interface is stored in the electronic device, the electronic device can determine the status information of at least one resource receiving interface by using the resource registration request.

S252, the resource receiving port whose status information is in a normal status is collected.

The electronic equipment collects the resource receiving ports with the state information of normal state from the state information of all the resource receiving ports so as to carry out data push of normal mode on the resource receiving ports.

And S253, pushing the real-time data to the resource receiving ports in the normal state in parallel by using at least one thread.

The electronic device may create a corresponding number of threads according to the number of the resource receiving ports in the normal state, that is, the created threads correspond to the resource receiving ports in the normal state one to one. After the threads are created, the electronic equipment pushes the real-time data to the resource receiving ports in the normal state by using the corresponding threads, so that the parallel pushing of the real-time data is realized.

S254, judging whether the pushing of the preset thread is abnormal.

And the electronic equipment judges whether the pushing of each thread is abnormal in real time in the data pushing process of each thread. Taking a preset thread as an example, the electronic device determines in real time whether pushing of the preset thread is abnormal, where the abnormality may be whether pushing of real-time data is successful, and when pushing of the thread is abnormal, S255 is executed; otherwise, executing S254 during the pushing process of the real-time data, and determining whether a pushing abnormal condition occurs.

And S255, stopping the real-time data pushing of the preset thread and setting the state information of the resource receiving port corresponding to the preset thread as retry.

When the electronic equipment determines that the real-time data pushing of the preset thread is abnormal, the data pushing of the preset thread is stopped, and the state information of the resource receiving port corresponding to the preset thread is set to retry.

If the resource receiving port a of the external system 1 and the resource receiving port b of the external system 2 are normal resource receiving ports, the electronic device creates 2 threads, such as thread 1 and thread 2, and transmits real-time data to the resource receiving ports a and b by using the thread 1 and thread 2, respectively. If the electronic equipment determines that the data push of the thread 1 is abnormal and the data push of the thread 2 is normal in the real-time data sending process, the electronic equipment sets the state information of the resource receiving port a corresponding to the thread 1 as retry and stops the data push of the thread 1. Thread 2 still normally pushes real-time data to resource receiving port b.

And S256, based on the resource receiving port in the retry state, performing data pushing corresponding to the retry mode.

After setting the state information of the resource receiving port corresponding to the preset thread as retry, the electronic device can push data corresponding to the retry mode.

The data transmission efficiency can be improved by pushing the data to the resource receiving ports in the normal state in parallel; and when the preset thread pushing is abnormal, setting the state information of the resource receiving port as retry, and pushing data in a retry mode to the resource receiving port, so that the normal state flow to the retry is realized to ensure the normal pushing of the data.

Specifically, the above S256 may include the following steps, and it is also understood that the data push in the retry mode may include the following steps:

(1) and the other thread pushes the current push failure data to the resource receiving port in the retry state again.

Continuing with the above embodiment, for the resource receiving port a, the electronic device opens another thread to push the currently failed-to-push data to the resource receiving port a again.

(2) And judging whether the current push failure data is successfully pushed to the resource receiving port in the retry state within the preset condition.

The preset condition may be a preset number of times, a preset time interval, or the like, and may be specifically set according to an actual situation, which is not limited herein.

When the current push failure data is successfully pushed to the resource receiving port in the retry state within the condition, executing the step (3); otherwise, step (5) is executed. For example, the electronic device may push, by using another thread, data that has failed in current push for at most 3 times to a resource receiving port in a retry state, and if all 3 times of push fails, execute step (5); otherwise, executing step (3).

(3) And setting the state information of the resource receiving port in the retry state as recovery.

In the retry process, if the data push failed at present is successful, the electronic device sets the state information of the resource receiving port in the retry state to be in recovery.

If the data push is successful in the retry mode, the state information of the resource receiving port is set to be in recovery, so that the state flow from retry to recovery is realized, and the guarantee is provided for the data push.

(4) And carrying out data push corresponding to the recovery mode based on the recovery resource receiving port.

Specifically, the step (4) may include the following steps, and it is also understood that the data pushing manner of the recovery mode includes the following steps:

4.1) extracting the configuration information of the resource receiving port in the recovery process.

The configuration information may include a flag indicating that the resource receiving port pushes the interrupted data, and when the flag indicating that the resource receiving port needs to push the interrupted data is 1, historical data from the interrupted position to the current position needs to be pushed to the resource receiving port in recovery; when the flag bit 0 of the interrupt data is pushed, it indicates that the resource receiving port does not need to push the interrupted data.

4.2) judging whether the interrupted data needs to be pushed or not based on the configuration information.

When the interrupted data needs to be pushed, executing 4.3); otherwise, directly pushing real-time data to the resource receiving port in recovery.

4.3) reading the historical data from the interrupt position to the current position.

When a breakpoint transmission is required, the electronic device can create a child listener. The sub-listener is created when the state is recovery according to the configuration information of the resource registration request of the external system, wherein the sub-listener is required to be created when the breakpoint returns. The sub-listener is a method for acquiring historical data from the interruption point of the resource source to the latest real-time point.

And 4.4) pushing the historical data to a resource receiving port in recovery, and judging whether pushing failure occurs in the pushing process.

When the pushing failure does not occur, executing the step 4.5); otherwise, step 4.7) is performed.

4.5) setting the state information of the resource receiving port in the recovery to be normal.

And when the electronic equipment successfully pushes the historical data to the recovered resource receiving port, setting the state information of the recovered resource receiving port as normal.

4.6) carrying out data push corresponding to the normal mode based on the resource receiving port in the normal state.

In the recovery mode, interruption continuous transmission can be carried out from the interruption position, so that data is not lost; and when the historical data pushing fails, the state of the resource receiving port is transferred from the recovery to the normal state, and the guarantee is provided for recovering the interrupted pushing data.

Please refer to the above S252 and the related description of the subsequent steps, which are not repeated herein.

4.7) setting the state information of the resource receiving port in the recovery as an exception, and pushing data corresponding to the exception mode based on the resource receiving port in the exception state.

When the electronic equipment is abnormal when pushing the historical data to the resource receiving port in recovery, the state information of the resource receiving port in recovery is set to be abnormal, and then data pushing corresponding to an abnormal mode is carried out on the basis of the resource receiving port in the abnormal state.

In the recovery mode, if the data push fails, the state of the resource receiving port is directly set to be abnormal, and retry is not performed, so that repeated retry of data transmission is avoided, and the real-time performance of data transmission is improved.

(5) The status of the resource receiving port in the retry state is set to abnormal.

When the electronic equipment fails to push the current push-failed data to the resource receiving port in the retry state, the state of the resource receiving port in the retry state is set to be abnormal. When the resource receiving port in the retry state fails to successfully push the current data failed to be pushed within the preset conditions, the state flow from the retry to the exception of the resource receiving port is realized, and the guarantee is provided for the normal pushing of the data.

(6) And carrying out data push corresponding to the abnormal mode based on the resource receiving port in the abnormal state.

Specifically, the step (6) may include a step, and it may also be understood that the data pushing in the abnormal mode includes the following steps:

6.1) updating the state of the resource receiving port receiving the abnormal state to normal information.

When determining that the resource receiving port is abnormal, the electronic device may send alarm information to the external system, so that the external system performs corresponding processing on the resource receiving port. After the external system completes the processing of the resource receiving port, status update information may be sent to the electronic device to request the status of the resource receiving port in the abnormal state to be updated to the normal state.

6.2) judging whether the state of the set resource receiving port in the abnormal state is abnormal or not.

After receiving information sent by an external system to update the state of the resource receiving port in the abnormal state to be normal, the electronic device first determines whether the state of the resource receiving port in the abnormal state stored locally is abnormal. When the set resource receiving port in the abnormal state is in the abnormal state, executing the step 6.3); otherwise, step 4.6) is performed.

6.3) setting the state information of the resource receiving port in the abnormal state as recovery.

When the electronic equipment determines that the state of the resource receiving port in the abnormal state stored locally is still abnormal, the electronic equipment firstly sets the state information of the electronic equipment to be recovered.

6.4) carrying out data push corresponding to the recovery mode based on the recovery resource receiving port.

Please refer to the description of step (4) above, which is not repeated herein.

In the data transmission method provided in this embodiment, when the state of the resource receiving port in the abnormal state is updated to be normal externally, the state of the resource receiving port is not directly set to be normal locally, but is set to be recovered first, and the abnormal state is transferred to the state in recovery, so that a guarantee is provided for normal transmission of data.

The internal state transition mechanism is shown in fig. 4, and the specific state transition rule is as follows:

(1) normal state to retry state

a) A plurality of external systems can register the same resource type, and after the internal monitor acquires data, a plurality of main processes are started in parallel to push the data.

b) When the data pushing is abnormal, the state of the external system resource receiving port with the abnormal data pushing is changed into retry (retry).

c) The main flow stops pushing new real-time data to the resource receiving port of the current external system.

d) The initiator sub-process retries push and proceeds to step (2).

(2) Retry state to exception state

a) The retry of the resource reception ports of the plurality of external systems may be performed in parallel with the plurality of sub-processes.

b) And three times of retries are carried out to push data to the resource receiving port of the current external system.

c) When the exception still occurs in the three retries, the state of the resource receiving port is changed to be an exception (exception).

d) The retry sub-flow is closed.

(3) Retrying state to recovering state

a) And three times of retries are carried out to push data to the resource receiving interface of the current external system.

b) When any one time of pushing data is successful, retry is stopped, and the state of the resource receiving interface is changed to be recovering (resetting).

c) The retry sub-flow is closed.

(4) Abnormal to recovering state

a) After the external system is started, the information for updating the state of the resource receiving port to be normal is sent to the electronic equipment

b) When the state of the resource receiving port received by the electronic equipment is to be changed to be normal, whether the state of the resource receiving port in the current database is normal or not is checked.

c) If the resource receiving port in the database is in a normal state, no processing is performed.

d) If the resource receiving port in the database is in an abnormal state, the resource receiving port is not updated to a normal state first and is updated to a recovery state first (restore).

e) And (5) starting a recovery flow and entering steps (5) and (6).

(5) Restoring an in-progress state to a retry state

a) And reading the configuration information of the resource receiving port with the state of all the resource receiving ports being recovered (restore).

b) And starting a recovery sub-process to push the interrupted data according to whether the interrupted data needs to be recovered.

c) When the interrupted data is pushed, the data is abnormal and no longer retried, the state of the resource receiving port is immediately changed into abnormal (anomaly), and the interruption data recovery sub-process is closed, wherein the sub-process comprises a sub-listener.

d) And (5) repeating the step.

(6) Restoring the in-process state to the normal state

b) The interrupted data is pushed as needed.

c) And if all the interrupted data are pushed without exception, changing the state to normal (normal).

d) The recovery flow is closed.

In this embodiment, a data transmission method is provided, which can be used in the above-mentioned electronic devices, such as a server, a computer, a tablet computer, and the like, and fig. 5 is a flowchart of the data transmission method according to the embodiment of the present invention, as shown in fig. 5, the flowchart includes the following steps:

and S31, providing the locally opened resource types.

The resource type corresponds to a resource source.

Please refer to S21 in fig. 3 for details, which are not described herein.

S32, at least one resource registration request for the target resource type is received.

The resource registration request corresponds to the resource receiving port, wherein the resource registration request carries configuration information, and the configuration information comprises a data filtering condition of the resource receiving port.

Please refer to S22 in fig. 3 for details, which are not described herein.

S33, based on the resource registration request, determining the target resource source corresponding to the target resource type.

Please refer to S23 in fig. 3 for details, which are not described herein.

And S34, extracting the real-time data from the target resource source.

Please refer to S24 in fig. 3 for details, which are not described herein.

And S35, pushing the real-time data to at least one resource receiving port.

Specifically, the step S35 includes the following steps:

s351, filtering the real-time data by using the data filtering condition.

And the electronic equipment filters the extracted real-time data based on the data filtering condition in the configuration information of each resource registration request.

S352, pushing the filtered real-time data to at least one resource receiving port.

For the filtered real-time data, the electronic device can push the real-time data to the resource receiving ports according to the state information of the resource receiving ports. Please refer to S25 in fig. 3 for details, which are not described herein.

If the filtered real-time data is empty, pushing the data to the corresponding resource receiving port is not needed; if the filtered real-time data is not empty, the filtered real-time data needs to be pushed to the resource receiving ports according to the state information of the resource receiving ports.

According to the data transmission method provided by the embodiment, the real-time data is filtered before being transmitted, so that the transmitted real-time data can meet the external requirements.

As a specific implementation manner of the embodiment of the present invention, as shown in fig. 6, the data transmission method includes the following processing flows:

(1) resource mapping: the resource sources are mapped to resource types.

a) The electronic device needs to open data to the outside, and the electronic device will call an interface to bind the resource source and the resource type.

b) The resource mapping component will be responsible for creating the correspondence of the two.

(2) Resource type announcement: telling the external system which resource types are currently available to be used at registration time.

(3) Required resource registration and status update: and the external system performs registration and state updating operation according to the type of the required resource.

a) The external system acquires the authority of the electronic device.

b) The external system calls a registration interface opened by the electronic equipment to the outside, and provides the registered resource type, the resource receiving port information, the state of the resource receiving port (the state of a receiving address needs to be updated immediately after the external system is restarted every time), the push type name (which can be pushed in an HTTP form and can be expanded), the mark of whether the interrupted data needs to be pushed again, the number of interrupted messages to be pushed again every time and the data filtering condition. c) After the external system is restarted, the state of the resource receiving port needs to be updated by calling the state updating interface opened by the open platform to the outside. Resource registration requests sent by the external system are stored in a database and cached for use by the internal state flow mechanism.

(4) A resource listener: and monitoring the data source of the mapping according to the resource type registered by the external system, acquiring the required data from the resource source in real time, and sending the data to the filter for processing.

a) And detecting the type of the resource needing to be processed in the cache in a round-robin mode.

b) And creating respective listeners for the resource types specified by the external system, and monitoring the data sources mapped by the resource types by the listeners.

c) Once any new real-time data is monitored, the real-time data is sent to the filter for processing.

(5) A filter: unwanted real-time data is filtered according to the requirements of the external system.

a) And obtaining the configuration information of the external system from the cache.

b) And filtering the real-time data according to the filtering condition configured by the external system.

c) And if the real-time data needs to be pushed after being filtered, the real-time data is handed to data distribution processing.

d) If the real-time data does not need to be pushed after being filtered, the real-time data is discarded without being pushed.

(6) Data distribution processing: and encapsulating the data to be transmitted into data with a fixed structure, and transmitting the data to an internal state circulation mechanism for processing.

a) The real-time data is encapsulated into a message with a fixed format, which can be JSON, on the premise that the format of the communication data needs to be negotiated with an external system. The message is hash or encrypted as required to prevent tampering or pushed by adopting an https protocol, and also needs to be negotiated with an external system.

b) And acquiring all resource receiving port information of the external system needing the data of the current resource type from the cache.

c) All real-time data to be processed are classified and assembled into transmission data according to the resource receiving ports, and the transmission data are handed to an internal state transfer mechanism for processing.

(7) Internal state flow mechanism: the method is used for state transition in three conditions of normal push, abnormal retry and abnormal recovery. Converting rules: normal (normal) - > retry (retry) - > abnormal (analog) - > recovering (restore).

a) The data after the data distribution processing will have the information of the designated resource receiving port, and the state of the resource receiving port is filtered.

b) If the resource receiving port state is normal (normal), and the interface information that the resource receiving port state is normal (normal) is collected, the process proceeds to step (8).

c) If the resource receiving port status is abnormal (anomaly), discard and do not process.

d) If the resource receiving port state is recovering (restore), the process proceeds to step (11).

(8) Data push: and pushing the data to a resource receiving port designated by the external system.

a) And sending the messages to the resource receiving ports corresponding to the external systems in parallel by adopting multiple threads.

b) If any thread pushes an exception, step (9) is entered.

c) If there is no abnormality, the present process is ended, and the process proceeds to step (4) again.

(9) Abnormal retry: push information again for push with exception, try 3 times.

a) The resource receiving port in which the data is pushed abnormally is set to retry (retry). When the resource receiving port set as retry stops the real-time data push of the main line.

b) Another thread attempts to push the data that is currently failed to be sent to the resource receiving port of the external system again for 3 times.

c) If the process is successful, the state of the resource receiving port is set to recovery (restore), and the process proceeds to step (11).

d) If the attempt is failed, setting the state of the resource receiving port as an exception (exception), recording the data position of the exception interrupt, interrupting the process, and waiting for the external system to call state update for state recovery.

(10) Abnormal information: the administrator can view all the anomalous push information through the interface.

(11) And (3) recovering the exception: and recovering the state of the abnormal resource receiving port.

a) Reading the resource receiving port configuration information of all resource receiving port state being recovery

b) And (4) according to the configuration, whether the interrupted data needs to be pushed or not, and if so, entering the step (12).

c) If the interrupted data does not need to be pushed, the current real-time data is pushed, the pushing is successful, and the state of the resource receiving port is changed from recovery (restore) to normal (normal).

d) And destroying the current sub-line resource monitor so as to enable the common main line resource monitor to continue monitoring the real-time data.

(12) Breakpoint resuming: and interrupting the continuous transmission of the interrupted data according to the requirement.

a) Creating a sub-line resource listener;

b) reading corresponding data from the position of the abnormal interrupt;

c) pushing data to a resource receiving port of an external system according to the specified data volume pushed each time, and recording the position successfully pushed each time until the latest real-time data at present is tracked;

d) if an exception occurs in the process of pushing the interrupted data, entering the step (9);

e) and if the data from the interrupt position to the current real-time latest position is pushed completely and no exception occurs, changing the state of the resource receiving port from recovery (restore) to normal (normal).

The data transmission method provided by the embodiment has the beneficial effects that:

(1) different types of open data sources are abstracted into resource types, and an external system only knows the resource types but not the concrete internal data sources, so that the internal data sources are easy to switch, an external open interface is not required to be provided on an open platform, the external system cannot sense the change of the data sources, and the internal migration and expansion are convenient.

(2) The resource monitor of the electronic equipment monitors data in real time, provides a function of filtering the data, and distributes and pushes the data to a plurality of external systems in parallel, so that the request pressure of the external systems to the internal server is reduced, the data can be pushed in real time, the pushing is interrupted unless an interface is abnormal, the data interruption and continuous transmission can be selected, and the data is ensured not to be lost.

(3) The electronic equipment records abnormal information when the real-time pushed data is abnormal, so that an administrator can conveniently and quickly check abnormal problems so as to guide an external system and the electronic equipment to repair related problems.

(4) The internal state flow mechanism provides guarantee for abnormal retry and recovery of interrupted push data.

(5) This scheme is not tied to any software, thereby easily extending the software functionality. If an abstract resource type needs to be added, only the mapping between the data source and the resource type needs to be configured, and the real-time push related processing does not need to be concerned. If the current listener cannot process the resource type, a satisfactory listener can be realized by extension.

In this embodiment, a data transmission device is further provided, and the data transmission device is used to implement the foregoing embodiments and preferred embodiments, which have already been described and are not described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

The present embodiment provides a data transmission apparatus, as shown in fig. 7, including:

a providing module 41, configured to provide a locally open resource type, where the resource type corresponds to a resource source;

a receiving module 42, configured to receive at least one resource registration request for a target resource type, where the resource registration request corresponds to a resource receiving port;

a determining module 43, configured to determine, based on the resource registration request, a target resource source corresponding to the target resource type;

an extraction module 44, configured to extract real-time data from the target resource source;

and a pushing module 45, configured to push the real-time data to at least one resource receiving port.

The data transmission device provided by this embodiment determines the target resource source corresponding to the target resource type by using the corresponding relationship between the resource type and the resource source, that is, the real-time data can be extracted from the target resource source and pushed to the resource receiving port, so as to achieve the purpose of pushing the data in real time.

The data transmission means in this embodiment is presented in the form of functional units, where a unit refers to an ASIC circuit, a processor and a memory executing one or more software or fixed programs, and/or other devices that may provide the above-described functionality.

Further functional descriptions of the modules are the same as those of the corresponding embodiments, and are not repeated herein.

An embodiment of the present invention further provides an electronic device, which has the data transmission apparatus shown in fig. 7.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to an alternative embodiment of the present invention, and as shown in fig. 8, the electronic device may include: at least one processor 51, such as a CPU (Central Processing Unit), at least one communication interface 53, memory 54, at least one communication bus 52. Wherein a communication bus 52 is used to enable the connection communication between these components. The communication interface 53 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 53 may also include a standard wired interface and a standard wireless interface. The Memory 54 may be a high-speed RAM Memory (volatile Random Access Memory) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 54 may alternatively be at least one memory device located remotely from the processor 51. Wherein the processor 51 may be in connection with the apparatus described in fig. 7, the memory 54 stores an application program, and the processor 51 calls the program code stored in the memory 54 for performing any of the above-mentioned method steps.

The communication bus 52 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 52 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 8, but this is not intended to represent only one bus or type of bus.

The memory 54 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (english: non-volatile memory), such as a flash memory (english: flash memory), a hard disk (english: hard disk drive, abbreviated: HDD) or a solid-state drive (english: SSD); the memory 54 may also comprise a combination of the above types of memories.

The processor 51 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP.

The processor 51 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.

Optionally, the memory 54 is also used to store program instructions. The processor 51 may call program instructions to implement the data transfer method as shown in the embodiments of fig. 2, 3 and 5 of the present application.

An embodiment of the present invention further provides a non-transitory computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions may execute the data transmission method in any of the above method embodiments. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method of data transmission, comprising:

extracting real-time data from the target resource source;

and pushing the real-time data to at least one resource receiving port.

2. The method of claim 1, wherein said pushing said real-time data to at least one of said resource receiving ports comprises:

3. The method according to claim 2, wherein said pushing the real-time data to at least one of the resource receiving ports according to the status information of at least one of the resource receiving ports comprises:

judging whether the pushing of the preset thread is abnormal or not;

or the like, or, alternatively,

4. The method of claim 3, wherein the resource receiving port based on the retry state performs data pushing corresponding to a retry mode, comprising:

or the like, or, alternatively,

5. The method according to claim 2, wherein said pushing the real-time data to at least one of the resource receiving ports according to the status information of at least one of the resource receiving ports comprises:

6. The method according to claim 4 or 5, wherein the resource registration request further carries configuration information, and the performing data pushing corresponding to the recovery mode based on the resource receiving port in recovery includes:

7. The method of claim 6, wherein the pushing data corresponding to the recovering mode based on the recovering resource receiving port comprises:

8. The method according to claim 2, wherein said pushing the real-time data to at least one of the resource receiving ports according to the status information of at least one of the resource receiving ports comprises:

9. The method according to claim 4, 7 or 8, wherein the resource receiving port based on the abnormal state performs data pushing corresponding to an abnormal mode, comprising:

10. The method according to claim 2, wherein the resource registration request carries configuration information, the configuration information includes a data filtering condition, and the pushing the real-time data to at least one of the resource receiving ports according to the status information of at least one of the resource receiving ports comprises:

filtering the real-time data by using the data filtering condition;

11. A data transmission apparatus, comprising:

a receiving module, configured to receive a resource registration request for at least one target resource type, where the resource registration request corresponds to a resource receiving port;

12. An electronic device, comprising:

a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the data transmission method of any one of claims 1-10.

13. A computer-readable storage medium storing computer instructions for causing a computer to perform the data transmission method of any one of claims 1 to 10.