CN112422613B - Data processing method, data processing platform and computer readable storage medium - Google Patents
Data processing method, data processing platform and computer readable storage medium Download PDFInfo
- Publication number
- CN112422613B CN112422613B CN202010974990.4A CN202010974990A CN112422613B CN 112422613 B CN112422613 B CN 112422613B CN 202010974990 A CN202010974990 A CN 202010974990A CN 112422613 B CN112422613 B CN 112422613B
- Authority
- CN
- China
- Prior art keywords
- data
- nodes
- data processing
- node
- engine module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/03—Protocol definition or specification
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/18—Multiprotocol handlers, e.g. single devices capable of handling multiple protocols
Abstract
The embodiment of the invention provides a data processing method, a data processing platform and a computer readable storage medium. The data processing method according to the embodiment of the invention comprises the following steps: in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol; dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
Description
Technical Field
The present application relates to the field of computer data processing, and in particular, to a data processing method, a data processing platform, and a computer-readable storage medium.
Background
The data processing platform can be used for carrying out various related processes such as collection, analysis, forwarding, storage and the like on a large number of quantities. With the continuous expansion of computer applications, data processing platforms are also being widely used.
Generally, after a data processing platform is built, the architecture and functions performed by the components are fixed. If in the subsequent data processing process, it is desired to introduce data in a new format or a new transmission protocol, further customized development and function deployment in a targeted manner need to be performed on the data processing platform, and the operation of the data processing platform needs to be suspended for updating, thereby resulting in a reduction in data processing efficiency of the data processing platform and a long time consumption. In addition, the data processing platform with a fixed architecture cannot perform dynamic allocation and function adjustment on each node for data processing, and cannot realize high-availability management of the nodes.
Disclosure of Invention
To solve the above technical problem, according to an aspect of the present invention, there is provided a data processing method applied to a data processing platform, including: in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol; dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
Optionally, the at least one data comprises: at least one of text data, picture data, audio data, video data, and XML format data; the at least one transport protocol includes: at least one of TCP, UDP, HTTP, FTP, WebService, MQ and serial port transmission protocol.
Optionally, in the data processing process, before acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol, the method further includes: constructing the data processing platform according to the pre-estimated parameters; and carrying out data processing by using the constructed data processing platform.
Optionally, the data processing engine module corresponding to at least one data or at least one transport protocol is a data processing engine module developed through an API interface template.
Optionally, dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform, so that the one or more nodes perform the configured functions based on the corresponding data or transmission protocol includes: transmitting the data processing engine module to the one or more nodes connected with the data processing platform to cause the one or more nodes to perform the configured functions based on the respective data or transmission protocol.
Optionally, dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform, so that the one or more nodes perform the configured functions based on the corresponding data or transmission protocol includes: configuring the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, wherein the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol.
Optionally, configuring the one or more nodes as a data collection node and a data forwarding node respectively based on the data processing engine module includes: and configuring the data acquisition node to monitor the execution state, the acquired data volume and the execution duration of the data acquisition function in real time and control the start and the end of the data acquisition function.
Optionally, configuring the one or more nodes as a data collection node and a data forwarding node respectively based on the data processing engine module includes: and the data forwarding node is configured to receive the data acquired by the data acquisition node through Kafka and forward the received data to a third-party application.
Optionally, configuring the one or more nodes as a data collection node and a data forwarding node respectively based on the data processing engine module includes: and configuring the data forwarding node as a rule based on a rule loaded by a rule engine, processing the data to be forwarded, and forwarding the processed data to a third-party application.
Optionally, configuring the one or more nodes as a data collection node and a data forwarding node respectively based on the data processing engine module further includes: and switching one or more data acquisition nodes into data forwarding nodes and/or switching one or more data forwarding nodes into data acquisition nodes according to the function execution condition of one or more nodes.
Optionally, dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform, so that the one or more nodes perform the configured functions based on the corresponding data or transmission protocol includes: grouping the one or more nodes connected with the data processing platform to obtain one or more node groups, respectively determining a main node and a backup node in the one or more node groups, and executing the configured function by using the determined main node.
Optionally, dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform, so that the one or more nodes perform the configured functions based on the corresponding data or transmission protocol, further includes: when a primary node performing the configured function fails, performing the function being performed by the primary node using a corresponding backup node located in the same node group as the primary node.
According to another aspect of the present invention, there is provided a data processing platform comprising: the data processing system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is configured to acquire a data processing engine module corresponding to at least one data or at least one transmission protocol in the data processing process; a configuration unit configured to dynamically configure, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
According to another aspect of the present invention, there is provided a data processing platform comprising: a processor; and a memory having computer program instructions stored therein, wherein the computer program instructions, when executed by the processor, cause the processor to perform the steps of: in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol; dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
According to another aspect of the invention, there is provided a computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the steps of: in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol; dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
According to the data processing method, the data processing platform and the computer readable storage medium of the invention, when data in a new format or a new transmission protocol is introduced, corresponding functions can be introduced into the data processing platform by acquiring the corresponding data processing engine module in the data processing process, and the functions of the nodes in the data processing platform can be dynamically configured without stopping the operation process of the data processing platform. The data processing method, the data processing platform and the computer readable storage medium can effectively improve the compatibility of the data processing platform, so that the data processing platform can be suitable for data processing of heterogeneous, multi-protocol and multi-data sources. In addition, by the data processing method, the data processing platform and the computer readable storage medium, the node functions in the data processing platform can be dynamically configured, so that high availability management of the nodes is realized, the data processing efficiency is improved, and the processing time is saved.
Drawings
The above and other objects, features, and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention when taken in conjunction with the accompanying drawings.
FIG. 1 illustrates a flow diagram of a data processing method applied to a data processing platform according to one embodiment of the present invention;
FIG. 2 illustrates an example of secondary development of a current data processing engine module through an API interface template, according to one embodiment of the present invention;
FIG. 3 illustrates an example architecture of a data processing platform according to one embodiment of the present invention;
FIG. 4 illustrates a block diagram of a data processing platform, according to one embodiment of the present invention;
FIG. 5 illustrates a block diagram of a data processing platform, according to one embodiment of the invention.
Detailed Description
A data processing method, a data processing platform, and a computer-readable storage medium according to embodiments of the present invention will be described below with reference to the accompanying drawings. In the drawings, like reference numerals refer to like elements throughout. It should be understood that: the embodiments described herein are merely illustrative and should not be construed as limiting the scope of the invention.
A data processing method applied to a data processing platform according to an embodiment of the present invention will be described below with reference to fig. 1. Fig. 1 shows a flow chart of the data processing method 100.
As shown in fig. 1, in step S101, during data processing, a data processing engine module corresponding to at least one data or at least one transmission protocol is acquired.
In the embodiment of the present invention, the data processing platform may be constructed on the basis of various estimated parameters. For example, the data scale growth condition of the data processing platform may be estimated, that is, the data processing platform may be constructed in consideration of estimation parameters such as daily data growth, data retention time, data redundancy, and the like. Further, for example, the data processing platform may be constructed in consideration of the requirement that the data processing platform satisfies the prediction parameters of Query Per Second (QPS), Transaction Per Second (TPS), and the like. For another example, the data processing platform may also be required to consider the following estimated parameters: the node planning method can be constructed according to the node usage, the node number, the CPU core number, the memory size, the disk size, the network bandwidth and the like of a specific service scene so as to meet the node planning. Alternatively, various control or management systems may be installed and certain commonly used data types or transport protocols, etc. may be configured when building a data processing platform. For example, installation of JDK for JAVA, installation of Zookeeper cluster for distributed application coordination, installation of Kafka cluster implementing message forwarding based on Zookeeper, installation of FastDFS cluster implementing distributed storage, installation of Redis for implementing cache management, and installation of MySql for configuration, etc. may be performed when the data processing platform is built. After the above-described build and configuration process is completed, data processing may begin using the data processing platform. Of course, the pre-estimated parameters, configuration and installation contents for constructing the data processing platform are only examples, and in practical application, the data processing platform can be configured individually according to a specific use scenario to meet the corresponding data processing requirement.
Optionally, the data processing platform may include one or more nodes, configured to execute functions configured by the data processing platform for the respective nodes. For example, the data processing platform may include one or more data collection nodes for data collection; for another example, the data processing platform may further include one or more data forwarding nodes, configured to forward the data collected by the data collection node to a third-party application. Of course, the nodes of the data processing platform may also be used to perform various other functions, such as storage, computation, retrieval, and so forth. The configuration functions of the above nodes are merely examples and are not limited herein.
Subsequently, if it is desired to introduce data in a new format or a new transmission protocol on the basis of the constructed data processing platform during the data processing, relevant operations can be performed during the data processing of the data processing platform by acquiring a data engine module corresponding to the data or the transmission protocol. Optionally, the at least one data may comprise: at least one of text data, picture data, audio data, video data, and XML format data; the at least one transport protocol may include: at least one of TCP, UDP, HTTP, FTP, WebService, MQ and serial port transmission protocol. Correspondingly, the introduced data source can also be at least one of a file, a database, a message system and a network; the introduction mode can also be a real-time mode and/or a timing mode and the like.
Optionally, the data processing engine module corresponding to the at least one data or the at least one transmission protocol may be a data processing engine module obtained after secondary development is performed on the basis of the current data processing engine module, for example, the data processing engine module may be developed through an API interface template. FIG. 2 illustrates an example of secondary development of a current data processing engine module through an API interface template according to one embodiment of the present invention. As shown in fig. 2, a data processing engine module corresponding to a certain data or a certain transmission protocol, which is developed secondarily, may specify various functions of starting, stopping, pausing, repeating, ending, result obtaining, input and output, and the like of a task. After the secondary development is completed, the acquired data processing engine module corresponding to a certain data or a certain transmission protocol can be packaged and uploaded to the data processing platform.
In step S102, dynamically configuring functions of one or more nodes in the data processing platform based on the data processing engine module, so that the one or more nodes execute the configured functions based on corresponding data or transmission protocols.
Optionally, after packaging and uploading the data processing engine module obtained by the secondary development to the data processing platform, the data processing platform may transmit the data processing engine module to the one or more nodes connected to the data processing platform, so that the one or more nodes execute the configured functions based on the corresponding data or transmission protocol. For example, the data processing platform may configure the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, and forward the data processing engine module obtained by the secondary development to the data acquisition node and the data forwarding node, so that the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol. Therefore, the operation can enable the data processing platform to be compatible with data in a new format or a new transmission protocol on the basis of not stopping the operation process of the data processing platform, further expands the data processing function of the data processing platform, realizes dynamic deployment of the data processing platform and flexible configuration of various application scenes, and conveniently and quickly realizes one-key data access of the data processing platform.
According to an embodiment of the present invention, the data collection node may be configured to monitor the execution state, the collected data amount, the execution duration of the data collection function, and control the start and the end of the data collection function in real time through a monitoring page at the front end. Subsequently, the data collected by the data collection node may be sent to the Kafka cluster to serve as a buffer for the collected data, taking into account the impact of the large amount of data on subsequent forwarding processing capabilities, thereby reducing the load on subsequent applications. In addition, the adoption of the Kafka cluster can ensure that the message cannot be lost and the data cannot be repeatedly consumed, and once the forwarding process fails, the Kafka cluster can be used for enabling the forwarding task to be continuously consumed from the last consumed place after being restarted, so that the repeated occupation of the system and the confusion of the data processing process are avoided.
The data forwarding node may then be configured to receive the data acquired by the data acquisition node via the pre-installed Kafka cluster and forward the received data to a third party application. Specifically, the data forwarding node may process (e.g., process, convert, analyze, etc.) the data to be forwarded based on a corresponding rule in a rule base loaded by the rule engine, and forward the processed data to the third-party application through the forwarder. In the process, the rules can be dynamically loaded through the rule engine, so that the corresponding rules can be applied to the data to be forwarded on the basis of not stopping various operations of the data processing platform, and the dynamic processing of the data is realized. Alternatively, the third party application may be, for example, a data governance, a shared platform, a real-time large screen, a decision analysis, and the like related application program.
According to an embodiment of the present invention, during the data collection and data forwarding process, the data processing platform may monitor the data collection and forwarding in real time to count the real-time data amount for each data collection and data forwarding task, which may include, for example, the data collection and/or data forwarding data amount counted every minute, hour, day, and week.
In addition, the data processing platform can also perform various processing on the acquired or forwarded data according to needs in the data acquisition and forwarding process, for example, the data can be compressed, encrypted and stored, so as to improve the security of the data and reduce the use space of the disk. Further, the data processing platform can also export and query the collected data through the management interface. The data processing condition can be known in real time in the data processing process by the data processing platform and the nodes in the data processing process, effective basis is provided for dynamically distributing the data processing nodes, and the data processing quantity and quality are improved.
According to an embodiment of the present invention, in order to implement load balancing of data processing, a data processing platform may reasonably allocate a task to a node with a lower load to execute according to an executed load condition (CPU, memory occupation condition, etc.), and may perform mutual adjustment for different functions executed by the node. Accordingly, the data processing platform may, depending on the functional performance of the one or more nodes, perform at least one of the following: switching one or more of the data collection nodes to a data forwarding node, switching one or more of the data forwarding nodes to a data collection node, starting or stopping a function performed by one or more of the data collection nodes, starting or stopping a function performed by one or more of the data forwarding nodes, and the like.
In addition, the data processing platform may further group the one or more nodes connected to the data processing platform to obtain one or more node groups, determine a master node and a backup node in the one or more node groups, respectively, and execute the configured function using the determined master node. Each node in the same node group can compete with the main and standby nodes through the Zookeeper, and the functions allocated to the node group can be executed only by the main node. On the basis, the backup node can monitor whether the main node is alive or not all the time, and if the main node fails or goes down, the backup node receives the notification and replaces the role of the main node to start and execute the tasks distributed to the original main node. The above-mentioned relevant functions of dynamically adjusting the data acquisition nodes and the data forwarding nodes, and dynamically allocating and monitoring the main and standby nodes in the same node group can realize the high availability management of the nodes, improve the data processing efficiency, and save the processing time.
According to the data processing method, when data in a new format or a new transmission protocol is introduced, corresponding functions can be introduced into the data processing platform by acquiring the corresponding data processing engine module in the data processing process, and the functions of the nodes in the data processing platform can be dynamically configured without stopping the operation process of the data processing platform. The data processing method can effectively improve the compatibility of the data processing platform, so that the data processing platform can be suitable for data processing of heterogeneous, multi-protocol and multi-data sources.
FIG. 3 illustrates an example architecture of a data processing platform according to one embodiment of the present invention. As shown in FIG. 3, the data processing platform includes a plurality of node groups, such as a node group consisting of primary node A and backup node A1, and a node group consisting of primary node B and backup node B1. The distributed node groups execute distributed function control through the Zookeeper cluster, and control the active-standby switching among the node groups through the Zookeeper cluster. The data processing platform allocates the functions to be executed for the respective node groups, in an example, the node group including the main node a and the backup node a1 may be allocated to execute the data collecting function, and the node group including the main node B and the backup node B1 may be allocated to execute the data forwarding function, or vice versa, which is not limited herein. Data provided by a data source, such as one or more of a file via TCP transport protocol, a database via Canel transport protocol, a network file via TCP/UDP transport protocol, an MQ via AMQP transport protocol, an FTP file via FTP transport protocol, etc., may be collected via a node group performing data collection functions (e.g., a node group including primary node a and backup node a 1), and the data collected by the data collection nodes may be received by Kafka cluster as a buffer for transmission to a node group performing data forwarding functions (e.g., a node group including primary node B and backup node B1) for forwarding to a third party application via a repeater. Specifically, the node group performing the data forwarding function may process (e.g., process, convert, parse, etc.) the data to be forwarded based on the corresponding rule in the rule base dynamically loaded by the rule engine, and forward the processed data to the third-party application through the forwarder. The third-party application can be, for example, a related application program such as data governance, a shared platform, a real-time large screen, decision analysis and the like.
In the example of fig. 3, if the data processing platform wishes to additionally introduce data or transmission protocols that are not currently configured, the configuration and deployment of the introduced data or transmission protocols may be implemented by performing secondary development on the corresponding data or transmission protocols by the current data processing engine module through, for example, an API interface template, and packaging and uploading the data or transmission protocols to the data processing platform.
For example, when a radar data acquisition task of a UDP protocol is desired to be introduced, a data processing engine module secondarily developed corresponding to the UDP transport protocol may be packaged and uploaded to a data processing platform for implementation. Optionally, a task related to the UDP transport protocol may be newly created on the data processing platform, and various parameters such as a task name, a node, real-time/timing, a data conversion method, a host, a port, whether compression is performed, an alarm interval, a data storage path, and the like may be set to start the task related to the UDP transport protocol. For another example, when a radar report data collection task of the FTP protocol is desired to be introduced, the data processing engine module secondarily developed corresponding to the FTP transmission protocol may be packaged and uploaded to the data processing platform for implementation. Optionally, a task related to the FTP transmission protocol may be newly created on the data processing platform, and various parameters such as a task name, a node, real-time/timing, a data conversion method, an address of the FTP server, a user name, a password, and a download path may be set to start the task related to the FTP transmission protocol. The operation can enable the data processing platform to be compatible with data in a new format or a new transmission protocol on the basis of not stopping the operation process of the data processing platform, further expands the data processing function of the data processing platform, realizes dynamic deployment of the data processing platform and flexible configuration of various application scenes, and conveniently and quickly realizes one-key data access of the data processing platform.
In the data processing process of the data processing platform, in order to realize load balancing of data processing, tasks can be distributed to nodes or node groups with lower loads to be executed, and functions can be adjusted. For example, the data processing platform in fig. 3 may start or stop a function executed by a node group that executes a data collection function, start or stop a function executed by a node group that executes a data forwarding function, switch a node group that executes a data collection function to a node group that executes a data forwarding function, or switch a node group that executes a data forwarding function to a node group that executes a data collection function, and so on.
In addition, the data processing platform may perform the corresponding functions of each node group only through the master nodes (e.g., master nodes a and B) in the node group. In this process, backup nodes A1 and B1 may monitor whether primary nodes A and B, respectively, are alive. For example, if the primary node a performing the data collection function fails or is down, the backup node a1 will receive the notification and replace the role of the primary node a, initiating and performing the corresponding data collection function. The above-mentioned relevant functions of dynamically adjusting the data acquisition nodes and the data forwarding nodes, and dynamically allocating and monitoring the main and standby nodes in the same node group can realize the high availability management of the nodes, improve the data processing efficiency, and save the processing time.
A data processing platform according to an embodiment of the invention is described below with reference to fig. 4. FIG. 4 shows a block diagram of a data processing platform 400 according to an embodiment of the present invention. As shown in fig. 4, data processing platform 400 includes an acquisition unit 410 and a configuration unit 420. In addition to these elements, data processing platform 400 may include other components, however, since these components are not relevant to the context of embodiments of the present invention, their illustration and description are omitted here. Furthermore, since the specific details of the following operations performed by the data processing platform 400 according to an embodiment of the present invention are the same as those described above with reference to fig. 2, a repeated description of the same details is omitted herein to avoid repetition.
The obtaining unit 410 of the data processing platform 400 in fig. 4 obtains a data processing engine module corresponding to at least one data or at least one transmission protocol during data processing.
In the embodiment of the present invention, the data processing platform may be constructed on the basis of various estimated parameters. For example, the data scale growth condition of the data processing platform may be estimated, that is, the data processing platform may be constructed in consideration of estimation parameters such as daily data growth, data retention time, data redundancy, and the like. Further, for example, the data processing platform may be constructed in consideration of the requirement that the data processing platform satisfies the prediction parameters of Query Per Second (QPS), Transaction Per Second (TPS), and the like. For another example, the data processing platform may also be required to consider the following estimated parameters: the node planning method can be constructed according to the node usage, the node number, the CPU core number, the memory size, the disk size, the network bandwidth and the like of a specific service scene so as to meet the node planning. Alternatively, various control or management systems may be installed and certain commonly used data types or transport protocols, etc. may be configured when building a data processing platform. For example, installation of JDK for JAVA, installation of Zookeeper cluster for distributed application coordination, installation of Kafka cluster implementing message forwarding based on Zookeeper, installation of FastDFS cluster implementing distributed storage, installation of Redis for implementing cache management, and installation of MySql for configuration, etc. may be performed when the data processing platform is built. After the above-described build and configuration process is completed, data processing may begin using the data processing platform. Of course, the pre-estimated parameters, configuration and installation contents for constructing the data processing platform are only examples, and in practical application, the data processing platform can be configured individually according to a specific use scenario to meet the corresponding data processing requirement.
Optionally, the data processing platform may include one or more nodes, configured to execute functions configured by the data processing platform for the respective nodes. For example, the data processing platform may include one or more data collection nodes for data collection; for another example, the data processing platform may further include one or more data forwarding nodes, configured to forward the data collected by the data collection node to a third-party application. Of course, the nodes of the data processing platform may also be used to perform various other functions, such as storage, computation, retrieval, and so forth. The configuration functions of the above nodes are merely examples and are not limited herein.
Subsequently, in the process of data processing, if it is desired to introduce data in a new format or a new transmission protocol on the basis of the constructed data processing platform, the obtaining unit 410 may perform related operations in the data processing process of the data processing platform by obtaining a data engine module corresponding to the data or the transmission protocol. Optionally, the at least one data may comprise: at least one of text data, picture data, audio data, video data, and XML format data; the at least one transport protocol may include: at least one of TCP, UDP, HTTP, FTP, WebService, MQ and serial port transmission protocol. Correspondingly, the introduced data source can also be at least one of a file, a database, a message system and a network; the introduction mode can also be a real-time mode and/or a timing mode and the like.
Optionally, the data processing engine module corresponding to the at least one data or the at least one transmission protocol may be a data processing engine module obtained after secondary development is performed on the basis of the current data processing engine module, for example, the data processing engine module may be developed through an API interface template. FIG. 2 illustrates an example of secondary development of a current data processing engine module through an API interface template according to one embodiment of the present invention. As shown in fig. 2, a data processing engine module corresponding to a certain data or a certain transmission protocol, which is developed secondarily, may specify various functions of starting, stopping, pausing, repeating, ending, result obtaining, input and output, and the like of a task. After the secondary development is completed, the acquired data processing engine module corresponding to a certain data or a certain transmission protocol can be packaged and uploaded to the data processing platform.
The configuration unit 420 dynamically configures functions of one or more nodes in the data processing platform based on the data processing engine module to cause the one or more nodes to perform the configured functions based on the corresponding data or transport protocol.
Optionally, after packaging and uploading the data processing engine module obtained by the secondary development to the data processing platform, the configuration unit 420 of the data processing platform may transmit the data processing engine module to the one or more nodes connected to the data processing platform, so that the one or more nodes execute the configured functions based on the corresponding data or transmission protocol. For example, the data processing platform may configure the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, and forward the data processing engine module obtained by the secondary development to the data acquisition node and the data forwarding node, so that the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol. Therefore, the operation can enable the data processing platform to be compatible with data in a new format or a new transmission protocol on the basis of not stopping the operation process of the data processing platform, further expands the data processing function of the data processing platform, realizes dynamic deployment of the data processing platform and flexible configuration of various application scenes, and conveniently and quickly realizes one-key data access of the data processing platform.
According to an embodiment of the present invention, the configuration unit 420 may configure the data acquisition node to be configured to monitor the execution state of the data acquisition function, the acquired data amount, the execution duration, and control the start and end of the data acquisition function in real time through a monitoring page at a front end. Subsequently, the data collected by the data collection node may be sent to the Kafka cluster to serve as a buffer for the collected data, taking into account the impact of the large amount of data on subsequent forwarding processing capabilities, thereby reducing the load on subsequent applications. In addition, the adoption of the Kafka cluster can ensure that the message cannot be lost and the data cannot be repeatedly consumed, and once the forwarding process fails, the Kafka cluster can be used for enabling the forwarding task to be continuously consumed from the last consumed place after being restarted, so that the repeated occupation of the system and the confusion of the data processing process are avoided.
Subsequently, the configuration unit 420 may configure the data forwarding node to receive the data acquired by the data acquisition node through the pre-installed Kafka cluster, and forward the received data to the third party application. Specifically, the data forwarding node may process (e.g., process, convert, analyze, etc.) the data to be forwarded based on a corresponding rule in a rule base loaded by the rule engine, and forward the processed data to the third-party application through the forwarder. In the process, the rules can be dynamically loaded through the rule engine, so that the corresponding rules can be applied to the data to be forwarded on the basis of not stopping various operations of the data processing platform, and the dynamic processing of the data is realized. Alternatively, the third party application may be, for example, a data governance, a shared platform, a real-time large screen, a decision analysis, and the like related application program.
According to an embodiment of the present invention, during the data collection and data forwarding process, the configuration unit 420 of the data processing platform may monitor the collection and forwarding of the data in real time to count the real-time data amount for each data collection and data forwarding task, which may include the data collection and/or data forwarding data amount counted every minute, hour, day, and week, for example.
In addition, the configuration unit 420 of the data processing platform may also perform various processing on the acquired or forwarded data as needed in the data acquisition and forwarding process, for example, may perform compression, encryption storage, and the like on the data, so as to improve the security of the data and reduce the usage space of the disk. Further, the data processing platform can also export and query the collected data through the management interface. The data processing condition can be known in real time in the data processing process by the data processing platform and the nodes in the data processing process, effective basis is provided for dynamically distributing the data processing nodes, and the data processing quantity and quality are improved.
According to an embodiment of the present invention, in order to implement load balancing of data processing, the configuration unit 420 of the data processing platform may reasonably allocate the task to the node with a lower load according to the load condition of execution (CPU, memory occupation condition, etc.), and may perform mutual adjustment for different functions executed by the node. Accordingly, the data processing platform may, depending on the functional performance of the one or more nodes, perform at least one of the following: switching one or more of the data collection nodes to a data forwarding node, switching one or more of the data forwarding nodes to a data collection node, starting or stopping a function performed by one or more of the data collection nodes, starting or stopping a function performed by one or more of the data forwarding nodes, and the like.
In addition, the configuration unit 420 of the data processing platform may further group the one or more nodes connected to the data processing platform to obtain one or more node groups, respectively determine a master node and a backup node in the one or more node groups, and perform the configured function by using the determined master node. Each node in the same node group can compete with the main and standby nodes through the Zookeeper, and the functions allocated to the node group can be executed only by the main node. On the basis, the backup node can monitor whether the main node is alive or not all the time, and if the main node fails or goes down, the backup node receives the notification and replaces the role of the main node to start and execute the tasks distributed to the original main node. The above-mentioned relevant functions of dynamically adjusting the data acquisition nodes and the data forwarding nodes, and dynamically allocating and monitoring the main and standby nodes in the same node group can realize the high availability management of the nodes, improve the data processing efficiency, and save the processing time.
According to the data processing platform, when data in a new format or a new transmission protocol is introduced, corresponding functions can be introduced into the data processing platform by acquiring the corresponding data processing engine module in the data processing process, and the functions of the nodes in the data processing platform can be dynamically configured without stopping the operation process of the data processing platform. The data processing method can effectively improve the compatibility of the data processing platform, so that the data processing platform can be suitable for data processing of heterogeneous, multi-protocol and multi-data sources.
A data processing platform according to an embodiment of the invention is described below with reference to fig. 5. FIG. 5 illustrates a block diagram of a data processing platform 500, according to an embodiment of the present invention. As shown in fig. 5, the data processing platform 500 may be a computer or a server.
As shown in fig. 5, data processing platform 500 includes one or more processors 510 and memory 520, although, of course, data processing platform 500 may also include input devices, output devices (not shown), and the like, which may be interconnected via a bus system and/or other form of connection mechanism. It should be noted that the components and configuration of the data processing platform 500 shown in FIG. 5 are exemplary only, and not limiting, and that the data processing platform 500 may have other components and configurations as desired.
The processor 510 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may utilize computer program instructions stored in memory 520 to perform desired functions, which may include: in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol; dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
Memory 520 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. On which one or more computer program instructions may be stored that may be executed by processor 510 to implement the functions of the data processing platform of embodiments of the present invention described above and/or other desired functions, and/or to perform a data processing method according to embodiments of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.
In the following, a computer readable storage medium according to an embodiment of the present invention is described, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the steps of: in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol; dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols.
Of course, the above-mentioned embodiments are merely examples and not limitations, and those skilled in the art can combine and combine some steps and apparatuses from the above-mentioned separately described embodiments to achieve the effects of the present invention according to the concepts of the present invention, and such combined and combined embodiments are also included in the present invention, and such combined and combined embodiments are not necessarily described herein.
Note that advantages, effects, and the like mentioned in the present invention are merely examples and not limitations, and they cannot be considered essential to various embodiments of the present invention. Furthermore, the foregoing detailed description of the invention is provided for the purpose of illustration and understanding only, and is not intended to be limiting, since the invention will be described in any way as it would be understood by one skilled in the art.
The block diagrams of devices, apparatuses, systems involved in the present invention are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".
The flowchart of steps in the present invention and the above description of the method are only given as illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to an element in the singular, for example, using the terms "a," "an," or "the" is not to be construed as limiting the element to the singular.
In addition, the steps and devices in the embodiments are not limited to be implemented in a certain embodiment, and in fact, some steps and devices in the embodiments may be combined according to the concept of the present invention to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.
The individual operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and/or software components and/or modules including, but not limited to, a circuit, an Application Specific Integrated Circuit (ASIC), or a processor.
The various illustrative logical blocks, modules, and circuits described may be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an ASIC, a field programmable gate array signal (FPGA) or other Programmable Logic Device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
The steps of a method or algorithm described in connection with the invention may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may reside in any form of tangible storage medium. Some examples of storage media that may be used include Random Access Memory (RAM), Read Only Memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, and the like. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. A software module may be a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.
The inventive methods herein comprise one or more acts for implementing the described methods. The methods and/or acts may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and/or use of specific actions may be modified without departing from the scope of the claims.
The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions on a tangible computer-readable medium. A storage media may be any available tangible media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. As used herein, disk (disc) includes Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc.
Accordingly, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions stored (and/or encoded) thereon that are executable by one or more processors to perform the operations described herein. The computer program product may include packaged material.
Software or instructions may also be transmitted over a transmission medium. For example, the software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, or microwave.
Further, modules and/or other suitable means for carrying out the methods and techniques described herein may be downloaded and/or otherwise obtained by a user terminal and/or base station as appropriate. For example, such a device may be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via storage means (e.g., RAM, ROM, a physical storage medium such as a CD or floppy disk) such that the user terminal and/or base station may obtain the various methods when coupled to or providing storage means to the device. Further, any other suitable technique for providing the methods and techniques described herein to a device may be utilized.
Other examples and implementations are within the scope and spirit of the invention and the following claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard-wired, or any combination of these. Features implementing functions may also be physically located at various locations, including being distributed such that portions of functions are implemented at different physical locations. Also, as used herein, including in the claims, "or" as used in a list of items beginning with "at least one" indicates a separate list, such that a list of "A, B or at least one of C" means a or B or C, or AB or AC or BC, or ABC (i.e., a and B and C). Furthermore, the word "exemplary" does not mean that the described example is preferred or better than other examples.
Various changes, substitutions and alterations to the techniques described herein may be made without departing from the techniques of the teachings as defined by the appended claims. Moreover, the scope of the present claims is not intended to be limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. Processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.
The previous description of the inventive aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the invention to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.
Claims (13)
1. A data processing method is applied to a data processing platform and comprises the following steps:
in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol;
dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols;
wherein dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform such that the one or more nodes perform the configured functions based on respective data or transport protocols comprises:
configuring the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, wherein the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol;
wherein configuring the one or more nodes as data collection nodes and data forwarding nodes, respectively, based on the data processing engine module further comprises:
according to the function execution condition of the one or more nodes, starting or stopping the function executed by the one or more data acquisition nodes, starting or stopping the function executed by the one or more data forwarding nodes, switching the one or more data acquisition nodes into the data forwarding nodes, and/or switching the one or more data forwarding nodes into the data acquisition nodes.
2. The method of claim 1, wherein,
the at least one type of data includes: at least one of text data, picture data, audio data, video data, and XML format data;
the at least one transport protocol includes: at least one of TCP, UDP, HTTP, FTP, WebService, MQ and serial port transmission protocol.
3. The method of claim 1, wherein during data processing, obtaining a data processing engine module corresponding to at least one of data or at least one transport protocol further comprises:
constructing the data processing platform according to the pre-estimated parameters;
and carrying out data processing by using the constructed data processing platform.
4. The method of claim 1, wherein,
the data processing engine module corresponding to at least one data or at least one transmission protocol is a data processing engine module developed through an API interface template.
5. The method of claim 1, wherein dynamically configuring, based on the data processing engine module, functionality of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functionality based on a respective data or transport protocol comprises:
transmitting the data processing engine module to the one or more nodes connected with the data processing platform to cause the one or more nodes to perform the configured functions based on the respective data or transmission protocol.
6. The method of claim 1, wherein configuring the one or more nodes as data collection nodes and data forwarding nodes, respectively, based on the data processing engine module comprises:
and configuring the data acquisition node to monitor the execution state, the acquired data volume and the execution duration of the data acquisition function in real time and control the start and the end of the data acquisition function.
7. The method of claim 1, wherein configuring the one or more nodes as data collection nodes and data forwarding nodes, respectively, based on the data processing engine module comprises:
and the data forwarding node is configured to receive the data acquired by the data acquisition node through Kafka and forward the received data to a third-party application.
8. The method of claim 1, wherein configuring the one or more nodes as data collection nodes and data forwarding nodes, respectively, based on the data processing engine module comprises:
and configuring the data forwarding node as a rule based on a rule loaded by a rule engine, processing the data to be forwarded, and forwarding the processed data to a third-party application.
9. The method of claim 1, wherein dynamically configuring, based on the data processing engine module, functionality of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functionality based on a respective data or transport protocol comprises:
grouping the one or more nodes connected with the data processing platform to obtain one or more node groups, respectively determining a main node and a backup node in the one or more node groups, and executing the configured function by using the determined main node.
10. The method of claim 9, wherein dynamically configuring, based on the data processing engine module, functionality of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functionality based on a respective data or transport protocol further comprises:
when a primary node performing the configured function fails, performing the function being performed by the primary node using a corresponding backup node located in the same node group as the primary node.
11. A data processing platform comprising:
the data processing system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is configured to acquire a data processing engine module corresponding to at least one data or at least one transmission protocol in the data processing process;
a configuration unit configured to dynamically configure, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols;
the configuration unit configures the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, wherein the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol;
the configuration unit starts or stops the functions executed by one or more data acquisition nodes, starts or stops the functions executed by one or more data forwarding nodes, switches one or more data acquisition nodes into data forwarding nodes, and/or switches one or more data forwarding nodes into data acquisition nodes according to the function execution conditions of one or more nodes.
12. A data processing platform comprising:
a processor;
and a memory having computer program instructions stored therein,
wherein the computer program instructions, when executed by the processor, cause the processor to perform the steps of:
in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol;
dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols;
wherein dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform such that the one or more nodes perform the configured functions based on respective data or transport protocols comprises:
configuring the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, wherein the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol;
wherein configuring the one or more nodes as data collection nodes and data forwarding nodes, respectively, based on the data processing engine module further comprises:
according to the function execution condition of the one or more nodes, starting or stopping the function executed by the one or more data acquisition nodes, starting or stopping the function executed by the one or more data forwarding nodes, switching the one or more data acquisition nodes into the data forwarding nodes, and/or switching the one or more data forwarding nodes into the data acquisition nodes.
13. A computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the steps of:
in the data processing process, acquiring a data processing engine module corresponding to at least one data or at least one transmission protocol;
dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform to cause the one or more nodes to perform the configured functions based on respective data or transport protocols;
wherein dynamically configuring, based on the data processing engine module, functions of one or more nodes in the data processing platform such that the one or more nodes perform the configured functions based on respective data or transport protocols comprises:
configuring the one or more nodes as a data acquisition node and a data forwarding node respectively based on the data processing engine module, wherein the data acquisition node performs a data acquisition function based on corresponding data or a transmission protocol, and the data forwarding node performs a data forwarding function based on corresponding data or a transmission protocol;
wherein configuring the one or more nodes as data collection nodes and data forwarding nodes, respectively, based on the data processing engine module further comprises:
according to the function execution condition of the one or more nodes, starting or stopping the function executed by the one or more data acquisition nodes, starting or stopping the function executed by the one or more data forwarding nodes, switching the one or more data acquisition nodes into the data forwarding nodes, and/or switching the one or more data forwarding nodes into the data acquisition nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010974990.4A CN112422613B (en) | 2020-09-16 | 2020-09-16 | Data processing method, data processing platform and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010974990.4A CN112422613B (en) | 2020-09-16 | 2020-09-16 | Data processing method, data processing platform and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112422613A CN112422613A (en) | 2021-02-26 |
| CN112422613B true CN112422613B (en) | 2022-02-01 |
Family
ID=74855355
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010974990.4A Active CN112422613B (en) | 2020-09-16 | 2020-09-16 | Data processing method, data processing platform and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112422613B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104345717A (en) * | 2014-10-17 | 2015-02-11 | 武汉华大优能信息有限公司 | Intelligent remote data acquisition system based on Internet of Things |
| CN105956082A (en) * | 2016-04-29 | 2016-09-21 | 深圳前海大数点科技有限公司 | Real-time data processing and storage system |
| CN105978887A (en) * | 2016-06-15 | 2016-09-28 | 晶赞广告（上海）有限公司 | Data access method, device and system for big data |
| CN106649496A (en) * | 2016-10-10 | 2017-05-10 | 国信优易数据有限公司 | Government affairs data collecting and sharing system and method |
| CN107302528A (en) * | 2017-06-12 | 2017-10-27 | 深圳市诺龙技术股份有限公司 | A kind of transmission method of multi-protocol data and a kind of gateway apparatus |
| CN109639467A (en) * | 2018-11-29 | 2019-04-16 | 华南理工大学 | Intelligent producing line multi-modal data interactive system and method based on SDN |
| CN110113318A (en) * | 2019-04-16 | 2019-08-09 | 深圳壹账通智能科技有限公司 | Front-end system data processing method, device, computer equipment and storage medium |
| CN111835786A (en) * | 2020-07-23 | 2020-10-27 | 杨承 | System for data acquisition and equipment control of multi-protocol equipment and implementation method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10798173B2 (en) * | 2017-08-18 | 2020-10-06 | Voko Solutions Limited | System and method for facilitating a data exchange amongst communication devices connected via one or more communication networks |
| CN110399089B (en) * | 2018-04-19 | 2023-05-05 | 阿里巴巴集团控股有限公司 | Data storage method, device, equipment and medium |
| CN109361532B (en) * | 2018-09-11 | 2021-08-24 | 上海天旦网络科技发展有限公司 | High availability system and method for network data analysis and computer readable storage medium |
| CN111352872A (en) * | 2020-02-20 | 2020-06-30 | 北京字节跳动网络技术有限公司 | Execution engine, data processing method, apparatus, electronic device, and medium |
-
2020
- 2020-09-16 CN CN202010974990.4A patent/CN112422613B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104345717A (en) * | 2014-10-17 | 2015-02-11 | 武汉华大优能信息有限公司 | Intelligent remote data acquisition system based on Internet of Things |
| CN105956082A (en) * | 2016-04-29 | 2016-09-21 | 深圳前海大数点科技有限公司 | Real-time data processing and storage system |
| CN105978887A (en) * | 2016-06-15 | 2016-09-28 | 晶赞广告（上海）有限公司 | Data access method, device and system for big data |
| CN106649496A (en) * | 2016-10-10 | 2017-05-10 | 国信优易数据有限公司 | Government affairs data collecting and sharing system and method |
| CN107302528A (en) * | 2017-06-12 | 2017-10-27 | 深圳市诺龙技术股份有限公司 | A kind of transmission method of multi-protocol data and a kind of gateway apparatus |
| CN109639467A (en) * | 2018-11-29 | 2019-04-16 | 华南理工大学 | Intelligent producing line multi-modal data interactive system and method based on SDN |
| CN110113318A (en) * | 2019-04-16 | 2019-08-09 | 深圳壹账通智能科技有限公司 | Front-end system data processing method, device, computer equipment and storage medium |
| CN111835786A (en) * | 2020-07-23 | 2020-10-27 | 杨承 | System for data acquisition and equipment control of multi-protocol equipment and implementation method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112422613A (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3637733B1 (en) | Load balancing engine, client, distributed computing system, and load balancing method | |
| CN108683720B (en) | Container cluster service configuration method and device | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| WO2017166643A1 (en) | Method and device for quantifying task resources | |
| CN113300881B (en) | 5G network-based scheduling method, device, equipment and storage medium | |
| CN110716744A (en) | Data stream processing method, system and computer readable storage medium | |
| CN111224806A (en) | Resource allocation method and server | |
| CN111966289B (en) | Partition optimization method and system based on Kafka cluster | |
| CN111125604B (en) | Page management method and device, terminal equipment and storage medium | |
| CN111147403B (en) | Message processing method and device, storage medium and electronic device | |
| CN110727697B (en) | Data processing method and device, storage medium and electronic device | |
| CN110297944B (en) | Distributed XML data processing method and system | |
| CN107948097B (en) | Bandwidth adjusting method and equipment | |
| CN106407636B (en) | Integration result statistical method and device | |
| CN112422613B (en) | Data processing method, data processing platform and computer readable storage medium | |
| EP3398304B1 (en) | Network service requests | |
| CN109308219B (en) | Task processing method and device and distributed computer system | |
| CN107371263B (en) | Method and device for scheduling uplink resources | |
| CN114546631A (en) | Task scheduling method, control method, core, electronic device and readable medium | |
| CN110874268B (en) | Data processing method, device and equipment | |
| CN112711587A (en) | Data processing method and device, electronic equipment and storage medium | |
| US11985072B2 (en) | Multimedia data stream processing method, electronic device, and storage medium | |
| Liu et al. | Cooper: Expedite Batch Data Dissemination in Computer Clusters with Coded Gossips | |
| WO2023020436A1 (en) | Network element data subscription method, device, and storage medium | |
| CN116886677A (en) | Data processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |