WO2021042840A1 - 数据处理方法、装置、服务器和计算机可读存储介质 - Google Patents

数据处理方法、装置、服务器和计算机可读存储介质 Download PDF

Info

Publication number
WO2021042840A1
WO2021042840A1 PCT/CN2020/098781 CN2020098781W WO2021042840A1 WO 2021042840 A1 WO2021042840 A1 WO 2021042840A1 CN 2020098781 W CN2020098781 W CN 2020098781W WO 2021042840 A1 WO2021042840 A1 WO 2021042840A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
message
message data
encoding
port
Prior art date
Application number
PCT/CN2020/098781
Other languages
English (en)
French (fr)
Inventor
陈友雄
谭仁棕
Original Assignee
深圳中兴网信科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳中兴网信科技有限公司 filed Critical 深圳中兴网信科技有限公司
Publication of WO2021042840A1 publication Critical patent/WO2021042840A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/547—Remote procedure calls [RPC]; Web services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/0001—Systems modifying transmission characteristics according to link quality, e.g. power backoff
    • H04L1/0015—Systems modifying transmission characteristics according to link quality, e.g. power backoff characterised by the adaptation strategy
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/2866—Architectures; Arrangements

Definitions

  • This application relates to the field of communication data transmission, such as a data processing method, a data processing device, a server, and a computer-readable storage medium.
  • one type of equipment in the case of processing multiple types of equipment, multiple protocols, and multiple encodings, one type of equipment is generally used to receive data to business applications, and one type of equipment is monitored, decoded, and analyzed separately, resulting in application system coding redundancy, Data cross is difficult to maintain, data is easy to lose, and resource consumption is high.
  • the first aspect of the embodiments of the present application provides a data processing method.
  • the second aspect of the embodiments of the present application provides a data processing device.
  • the third aspect of the embodiments of the present application provides a server.
  • a fourth aspect of the embodiments of the present application provides a computer-readable storage medium.
  • a data processing method including: configuring corresponding communication ports for the same type of networked devices, so that multiple networked devices of the same type can transmit message data through the communication ports, Among them, if there are multiple types of networked devices, a corresponding communication port is configured for each type of networked device; the message data monitored by at least one communication port is transferred to the corresponding encoding service through at least one message middleware , The message data is decoded by the encoding service to generate decoded data, wherein, before the message data is transferred to the corresponding encoding service of the message data, the encoding format of the message data is expanded.
  • a data processing device including: a memory, a processor, and a program stored in the memory and capable of running on the processor.
  • the program is executed by the processor, the implementation is as described above
  • the data processing method of any technical solution is provided.
  • a server including: the data processing device described in the technical solution of the second aspect.
  • Fig. 1 shows a schematic flowchart of a data processing method according to an embodiment of the present application
  • Fig. 2 shows a schematic diagram of a data processing method according to another embodiment of the present application
  • Fig. 3 shows a schematic block diagram of a data processing device according to an embodiment of the present application
  • Fig. 4 shows a schematic block diagram of a server according to an embodiment of the present application
  • Fig. 5 shows a schematic block diagram of a computer-readable storage medium according to an embodiment of the present application.
  • the data processing method includes:
  • Step 102 Configure corresponding communication ports for the same type of networked devices, so that multiple networked devices of the same type can transmit message data through the communication ports.
  • Step 104 The message data monitored by the at least one communication port is transferred to the corresponding encoding service of the message data through the at least one message middleware, and the encoding service decodes the message data to generate decoded data.
  • Step 106 Invoke the multi-thread pool service according to the data protocol of each networked device, and allocate the decoded data of each networked device to the execution thread in the multi-thread pool service for analysis, and generate analysis data.
  • this application proposes to configure a unified communication port (distributed data monitoring) for multiple networked devices of the same type, and the same type of networked devices are connected to the unified communication port For communication, it is logically separated from messages uploaded by other types of networked devices to ensure the atomicity of messages from multiple types of networked devices.
  • independent communication ports are set correspondingly, that is, in an environment where multiple types of networked devices exist, corresponding communication ports are configured for each type of networked device, and the message middleware is used to process multiple types of networked devices. Message data of multiple networked devices of various types.
  • one message middleware can be set for data processing, or multiple message middleware can be set for data processing.
  • the message data monitored by the communication port can be relayed and distributed through the message middleware, and messages in the same encoding format can be distributed.
  • the data is transferred to the corresponding encoding service for unified decoding, which improves the decoding efficiency and also facilitates the expansion of the encoding format of the message data.
  • the decoded message data will call the multi-threaded pool service according to the data protocol required by the networked device, and execute the analysis process through the multi-threaded pool service.
  • the thread pool management technology of the multi-threaded pool service supports parallel circulation, analysis and analysis of multiple data streams.
  • the completed message data is the device index data of the actual business meaning, which also realizes the transfer from the Internet of Things data to the intuitive interconnection data.
  • distributed data monitoring services, message decoding services, and multithreaded asynchronous analysis services can be abstracted as APIs for system calls.
  • the coding format, data protocol, data analysis field and type of message data can be configurable to access, so as to realize the compatibility of multi-type, multi-device, multi-coding, and multi-protocol high concurrency of networked devices.
  • corresponding communication ports are configured for the same type of networked devices, so that multiple networked devices of the same type can transmit message data through the communication ports, including: configuring a unified transmission for the same type of networked devices Control Protocol/Internet Protocol (Transmission Control Protocol/Internet Protocol, TCP/IP) port, so that multiple networked devices of the same type can transmit message data through the TCP/IP port, where the TCP/IP port is used to realize the Internet of Things and the Internet Docking.
  • TCP/IP Transmission Control Protocol/Internet Protocol
  • an independent TCP/IP port is configured for the same type of device, and the same type of device is connected to a designated port to achieve logical separation from messages uploaded by other types of devices, thereby ensuring the atomicity of messages from multiple types of devices.
  • Sex At the same time, the Internet of Things and the Internet are connected through the tcp/ip protocol of the TCP/IP port.
  • the socket-based connection method ensures the reliability of data message transmission.
  • the socket-based connection method ensures the reliability of data message transmission.
  • socket is a process communication mechanism based on BSD UNIX, used to describe the IP address and port, called a "socket", is a communication handle used to implement communication between different virtual machines or computing devices.
  • the host On the network, the host (server side) generally runs multiple applications and provides multiple services, each of which corresponds to an independent port. Client software connects to different ports through sockets to realize data interaction.
  • TCP/IP is called Transmission Control Protocol/Internet Protocol, in which network addresses are uniformly allocated, and each device and terminal in the network has a unique address.
  • the message data monitored by the communication port is transferred to the corresponding encoding service of the message data through at least one message middleware, including: after the message data is monitored by the communication port, through at least one based Asynchronous transmission principle is a message middleware that carries out parallel reception and distribution of data messages, and transfers message data to the corresponding encoding service of message data.
  • the message data monitored by the communication port is transferred to the corresponding encoding service of the message data through at least one message middleware, and the encoding service decodes the message data to generate the decoded data, including : After the communication port monitors the message data, it transfers the message data to the corresponding encoding service of the message data through the kafka message queue, and the encoding service decodes the message data to generate decoded data.
  • kafka is a high-throughput distributed publish-and-subscribe messaging middleware with high performance, durability, multiple copy backup, and horizontal expansion capabilities.
  • the Kafka message queue transfers message data with the same encoding format to the corresponding encoding service for decoding, which improves the compatibility of encoding formats and improves decoding efficiency.
  • Kafka message queue can process large amounts of data with high concurrency in real time, with low latency and good fault tolerance.
  • the message data monitored by the communication port is transferred to the corresponding encoding service of the message data through the message middleware, and the encoding service decodes the message data to generate the decoded data
  • the encoding format extension of the message data includes: parallel buffering the message data monitored by at least one communication port through at least one message middleware, according to the type of networked device Enqueue the corresponding message data into the message subject with the port number; after the message data enqueues successfully, the long connection of data transmission will be disconnected; when the message data is dequeued, the encoding format of the message data is extended, and according to the extended
  • the encoding format of the message data groups the message data; according to the grouped message data, the corresponding encoding service is called to decode the message data to generate decoded data, wherein the port number corresponds to the communication port one-to-one.
  • the message data monitored by the communication port is transferred to the corresponding encoding service through the message middleware, and the encoding service decodes the message data to generate the decoded data, including: using kafka
  • the message queue caches the message data in parallel, and queues the corresponding message data to the message subject with the port number according to the type of networked device; the long connection of data transmission is disconnected after the message data is successfully queued; when the message data is dequeued,
  • the MINA framework is called to expand the encoding format of the message data, and the message data is assembled and classified; according to the classification, the corresponding encoding service is called to decode the message data to generate decoded data.
  • the port number and communication The ports have a one-to-one correspondence.
  • assembling and classifying message data has the same meaning as grouping message data.
  • message data with the same encoding format can be grouped into the same group, and then a responding encoding service is invoked according to the encoding format of the group of message data to decode the group of message data to obtain decoded data.
  • the Kafka message queue caches the message data in parallel, the message data is queued to the corresponding message topic, and the message topic is connected with the port number of the communication port. After the message data enters the queue, the current tcp connection is disconnected, so as not to increase the system overhead due to long connections.
  • the MINA framework is called according to the port number to expand the coding format of the message data in the message subject, so as to realize the seamless connection between the Internet of Things and the Internet coding.
  • the traditional Internet of Things encoding format only supports American Standard Code for Information Interchange (ASCII) and UNICODE, and the encoding format of message data can be extended through the MINA framework to decode more data types .
  • MINA is an abstract, event-driven and asynchronous API based on TCP/IP and UDP/IP protocols derived from Java nio technology.
  • Extensible encoding formats include: Chinese Internal Code Specification (GBK), 8-bit Unicode Transformation Format (UTF-8), Unicode, Universal Semiotic Coding for Information Interchange, USCII) and 16-bit Unicode Transformation Format (UTF-16).
  • it further includes: detecting an error in the decoding process, using the MINA framework to call the write-back interface to return the error code to the networked device to which the message data of the error occurred in the decoding process, or detecting If an error occurs in the decoding process or the encoding does not conform to the encoding specification, the networked device to which the message data belongs to request a decoding error or the encoding does not conform to the encoding specification to retransmit the message.
  • the MINA framework is used to correct errors in the decoding process, so as to improve decoding accuracy and improve the reliability of the message data decoding process.
  • the multi-thread pool service is called according to the data protocol of each networked device, and the decoded data of each networked device is allocated to the execution thread in the multi-thread pool service for analysis
  • Generating analytical data includes: invoking the Executor thread pool framework according to the data protocol required by each networked device, and allocating the decoded data of each networked device to the execution threads of the Executor thread pool framework for analysis, and generating analytical data, where , The execution thread calls the analysis interface to point to the corresponding data protocol analysis method.
  • the Executor multi-thread management technology is used to implement data parallel analysis, which ensures the efficiency and timeliness of data analysis.
  • TCP messages generally include headers, authentication, encryption and decryption, message content, and so on. Since each type of networked device has a specific data standard, the data protocol is also different in different industries.
  • This application uses abstract interface genericization, expands and implements each type of data analysis through the same interface, and exposes the interface for invocation of different types of networked devices.
  • This application uses the Executor multi-thread management technology to manage threads, and each thread execution method calls the analysis interface to point to the corresponding analysis protocol method. And executor thread pool management technology supports parallel circulation and analysis of multiple data streams.
  • the data after the analysis is the device index data of the actual business meaning, which is equivalent to realizing the transfer from the Internet of Things data to the intuitive interconnection data.
  • Executor multi-thread management technology to realize data parallel analysis, ensuring the efficiency and timeliness of data analysis.
  • Executor is a management thread to realize thread reuse, reduce resource consumption, and improve response speed. This service thread is suitable for executing asynchronous tasks.
  • it further includes: storing, displaying, or forwarding the parsed data.
  • the parsed data is the parsed message data.
  • the parsed data can be used for data storage, platform display, forwarding of third-party applications, etc. according to actual business needs, so that IoT data can be uploaded to the Internet seamlessly Take advantage of it.
  • it further includes: performing distributed deployment of the port monitoring application service of the communication port to achieve load balancing.
  • load balancing is performed by means of distributed deployment of the monitoring application service of the communication port corresponding to the type of networked device , To ensure the stable and successful upload of networked device data.
  • the system performs load balancing by deploying port monitoring application services in a distributed manner to ensure the stable and successful upload of IoT device data.
  • the TCP/IP port uses Kafka to cache the received message data in parallel, joins the queue successfully and disconnects the current TCP/IP connection to avoid long connections that increase system overhead.
  • Messages of different IoT device types enter the queue into the message topic named with port number.
  • the MINA framework is called to expand the traditional data encoding format according to the port number on the topic, and then the message data is assembled and classified, and finally According to the classification, the codes of the same type are decoded uniformly. If a decoding error occurs, the MINA framework will call the write-back interface to return the error code to the device, prompting that the encoding does not conform to the encoding specification or the decoding error, and the device is required to retransmit the message.
  • TCP messages generally include headers, authentication, encryption and decryption, message content, and so on. Since each device type has a designated data standard, the data protocol is also different in different industries.
  • the solution uses abstract interfaces to generalize, expand and implement each type of data analysis through the same interface, and expose the interfaces for different types of devices to call.
  • Executor thread pool management technology supports parallel circulation and analysis of multiple data streams.
  • the data after the analysis is the device index data of the actual business meaning, which is equivalent to realizing the transfer from the Internet of Things data to the intuitive interconnection data.
  • the data transmission process mainly includes: classifying IoT devices according to the device types of the IoT devices, for each device type (for example, including: device type 1.
  • IoT devices of device type 2, device type 3, device type 4...device type n are respectively configured with corresponding TCP/IP ports (for example, including: listening port 1, listening port 2, listening port 3, listening port 4;Monitor port n), the message data monitored by the TCP/IP port is enqueued and dequeued through the Kafka message queue, and the encoding format is extended by the MINA framework and grouped according to the encoding format, and the message data is grouped according to the grouping situation Transfer to the corresponding decoding service (for example, including: decoding method 1, decoding method 2 and decoding method 3), and then the decoded message data is subjected to data analysis, and the Internet of Things data is converted into Internet data for data application.
  • n is a positive integer greater than or equal to 5.
  • This embodiment implements a technical solution for high-concurrency IoT data transmission interaction oriented to multiple device types, multiple frequencies, multiple protocols, and multiple encodings.
  • the embodiment is divided into three application subsystem services: distributed port monitoring, data decoding, and multi-threaded asynchronous analysis.
  • Distributed monitoring configures independent tcp ports for different types of devices. All devices of the same type use the same port for message transmission, and are transferred to different encoding services through kafka for message decoding.
  • the decoded data is allocated to an execution thread for analysis by calling the multi-threaded pool service according to the data protocol required by the device.
  • the parsed data is used for data storage, platform display, forwarding to third-party applications, etc., according to actual business needs.
  • API Application Programming Interface
  • the solution provides support for GBK, UTF-8, Unicode, USCII, UTF-16, etc. Decoding of 5 traditional IoT data encoding formats.
  • this embodiment also provides configurable access to the encoding format, data protocol, data analysis field and type. So as to realize the compatibility of multi-type, multi-device, multi-coding, and multi-protocol high concurrency for IoT devices.
  • a data processing device 300 includes: a memory 302, a processor 304, and a program stored in the memory 302 and running on the processor 304, and the program is executed by the processor 304 When realizing the data processing method as in any of the above embodiments.
  • a server 400 includes: the data processing device 300 described in the third embodiment.
  • the server 400 can realize: configure corresponding communication ports for the same type of networked devices, so that multiple networked devices of the same type can transmit message data through the communication ports; and transfer the message data monitored by the communication ports through the message middleware
  • the encoding service decodes the message data to generate decoded data; calls the multi-thread pool service according to the data protocol of each networked device, and distributes the decoded data of each networked device to multiple
  • the analysis is performed in the execution thread in the thread pool service, and the analysis data is generated.
  • a computer-readable storage medium 500 is also provided, on which a computer program 502 is stored, and the computer program 502 realizes the data defined in any of the above embodiments when executed. Approach.
  • the computer program 502 is executed when the computer program 502 is executed: configure corresponding communication ports for the same type of networked devices, so that multiple networked devices of the same type can transmit message data through the communication ports;
  • the message data monitored by a communication port is forwarded to the corresponding encoding service of the message data, and the encoding service decodes the message data to generate decoded data; according to the data protocol of each networked device, the multi-thread pool service is called to transfer the The decoded data of each networked device is allocated to the execution thread in the multi-thread pool service for analysis to generate analytical data.
  • a unified communication port for multiple networked devices of the same type connect the same type of networked devices to the unified communication port for communication, and logically separate the messages uploaded by other types of networked devices to ensure the atomicity of messages from various types of networked devices .
  • independent communication ports are correspondingly set for various types of networked devices, and the message middleware is used to process the message data of multiple types of multiple networked devices.
  • the message data monitored by the communication port is transferred and distributed through the message middleware, and the message data of the same encoding format is transferred to the corresponding encoding service for unified decoding, which improves the decoding efficiency and facilitates the expansion of the encoding format of the message data. .
  • the decoded message data will call the multi-threaded pool service according to the data protocol required by the networked device, and execute the analysis process through the multi-threaded pool service.
  • the thread pool management technology of the multi-threaded pool service supports parallel circulation, analysis and analysis of multiple data streams.
  • the completed message data is the device index data of the actual business meaning, which also realizes the transfer from the Internet of Things data to the intuitive interconnection data.
  • corresponding communication ports are configured for the same type of networked devices, so that multiple networked devices of the same type can transmit message data through the communication ports, including: configuring the same type of networked devices
  • a unified TCP/IP port enables multiple networked devices of the same type to transmit message data through the TCP/IP port, where the TCP/IP port is used to realize the connection between the Internet of Things and the Internet.
  • an independent TCP/IP port is configured for the same type of equipment, and the same type of equipment is logically separated from the messages uploaded by other types of equipment by connecting to the designated port, so as to ensure the atomicity of the messages of various types of equipment.
  • the Internet of Things and the Internet are connected through the tcp/ip protocol of the TCP/IP port.
  • the socket-based connection method ensures the reliability of data message transmission.
  • socket is a process communication mechanism based on BSD UNIX, used to describe the IP address and port, called a "socket", is a communication handle used to implement communication between different virtual machines or computing devices.
  • the host On the network, the host (server side) generally runs multiple applications and provides multiple services, each of which corresponds to an independent port. Client software connects to different ports through sockets to realize data interaction.
  • TCP/IP is called Transmission Control Protocol/Internet Protocol, in which network addresses are uniformly allocated, and each device and terminal in the network has a unique address.
  • the message data monitored by the communication port is transferred to the corresponding encoding service through the message middleware, and the encoding service decodes the message data to generate the decoded data, including : After the communication port monitors the message data, it transfers the message data to the corresponding encoding service through the kafka message queue, and the encoding service decodes the message data to generate decoded data.
  • Kafka is a high-throughput distributed publish-and-subscribe messaging middleware with high performance, durability, multiple copy backup, and horizontal expansion capabilities.
  • the Kafka message queue transfers message data with the same encoding format to the corresponding encoding service for decoding, which improves the compatibility of encoding formats and improves decoding efficiency.
  • Kafka message queue can process large amounts of data with high concurrency in real time, with low latency and good fault tolerance.
  • the message data monitored by the communication port is transferred to the corresponding encoding service through the message middleware, and the encoding service decodes the message data to generate the decoded data, including : Use the Kafka message queue to cache the message data in parallel, and queue the corresponding message data to the message topic with the port number according to the type of networked device; after the message data enters the queue successfully, the long data transmission connection is disconnected; the message data is out During the queue, according to the port number of the message subject, the MINA framework is called to expand the encoding format of the message data, and the message data is assembled and classified; according to the classification, the corresponding encoding service is called to decode the message data to generate the decoded data.
  • the port The number corresponds to the communication port one by one.
  • the Kafka message queue caches the message data in parallel, the message data is queued to the corresponding message topic, and the message topic is connected with the port number of the communication port. After the message data enters the queue, the current tcp connection is disconnected, so as not to increase the system overhead due to long connections.
  • the MINA framework is called according to the port number to expand the coding format of the message data in the message subject, so as to realize the seamless connection between the Internet of Things and the Internet coding.
  • the traditional IoT encoding format only supports ASCII and UNICODE, which can be extended through the MINA framework to decode more data types.
  • the computer program 502 according to any one of the above technical solutions, optionally, further includes: detecting an error in the decoding process, using the MINA framework to call the write-back interface to return the error code to the networked device to which the message data with the error occurred in the decoding process, or , It is detected that there is an error in the decoding process or that the encoding does not conform to the encoding specification, and the networked device to which the message data belongs to request a decoding error or the encoding that does not conform to the encoding specification to retransmit the message.
  • the MINA framework is used to correct errors in the decoding process, improve the decoding accuracy, and improve the reliability of the message data decoding process.
  • any one of the above technical solutions optionally, call the multi-thread pool service according to the data protocol of each networked device, and allocate the decoded data of each networked device to the execution thread in the multi-thread pool service Analyze in the process to generate analytical data, including: invoking the Executor thread pool framework according to the data protocol required by each networked device, and assigning the decoded data of each networked device to the execution threads of the Executor thread pool framework for analysis, and generating analysis Data, where the execution thread calls the parsing interface to point to the corresponding data protocol parsing method.
  • the Executor multi-thread management technology is used to realize data parallel analysis, which ensures the efficiency and timeliness of data analysis.
  • the computer program 502 according to any one of the above technical solutions, optionally, further includes: storing, displaying, or forwarding parsed data.
  • the parsed data is the parsed message data, and the parsed data can be used for data storage, platform display, forwarding third-party applications, etc. according to actual business needs, so that IoT data can be seamlessly uploaded to the Internet. Take advantage of it.
  • the computer program 502 according to any one of the above technical solutions, optionally, further includes: performing distributed deployment of the port monitoring application service of the communication port to achieve load balancing.
  • load balancing is performed by means of distributed deployment of the monitoring application service of the communication port corresponding to the type of networked device , To ensure the stable and successful upload of networked device data.
  • this application configures an independent port for a device type, and performs enqueue, decoding, and analysis according to the port allocation resources, so that different types of data do not interfere with each other.
  • the traditional coding can be extended by using the MINA framework. So as to realize the seamless connection between the Internet of Things and the interconnection code. Kafka realizes that the message enters the queue, that is, the TCP connection is disconnected, and the Executor is used for thread pool management to perform concurrent data processing to improve the utilization of server resources.
  • this application can be provided as methods, systems, or computer program products. Therefore, this application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this application may use one or more computer-usable storage media (including but not limited to disk storage, Compact Disc-Read Only Memory, CD-ROM), and optical storage containing computer-usable program codes. Etc.) in the form of a computer program product implemented on it.
  • computer-usable storage media including but not limited to disk storage, Compact Disc-Read Only Memory, CD-ROM
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • any reference signs located between parentheses should not be constructed as limitations on the claims.
  • the word “comprising” does not exclude the presence of parts or steps not listed in the claims.
  • the word “a” or “an” preceding a component does not exclude the presence of multiple such components.
  • the application can be realized by means of hardware including several different components and by means of a suitably programmed computer. In the unit claims that list several devices, several of these devices can be embodied by the same hardware item.
  • the use of the words first, second, and third, etc. do not indicate any order. These words can be interpreted as names.

Abstract

本申请提供了一种数据处理方法、装置、服务器、和计算机可读存储介质，其中，数据处理方法包括：为同一类型的联网设备配置对应的通信端口，以使同一类型的多个所述联网设备通过所述通信端口传输消息数据；通过至少一个消息中间件将至少一个所述通信端口监听到的所述消息数据转接至所述消息数据相应的编码服务，由所述编码服务对所述消息数据进行解码，生成解码数据，其中，将所述消息数据转接至所述消息数据相应的编码服务之前，对所述消息数据进行编码格式扩展。

Description

数据处理方法、装置、服务器和计算机可读存储介质
本申请要求在2019年09月02日提交中国专利局、申请号为201910846061.2的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及通讯数据传输领域，例如涉及一种数据处理方法、一种数据处理装置、一种服务器和一种计算机可读存储介质。
背景技术
随着4G通讯和5G通讯等通讯技术的快速发展，万物互联的物联网应用场景越来越多，在多个领域、行业和社会民生的方方面面都铺展开来，同时又存在互联设备类型，设备传输数据，设备厂家标准各不相同的窘况。面对大量不同类型设备，不同设备的传输数据格式各异，传输的编码千姿百态，让数据的接收、解码和解析过程变得困难重重，数据系统臃肿不堪。
相关技术中，对于处理多类型设备、多协议、多编码的情况，一般是通过一种类型设备从接收数据到业务应用，一种类型单独进行监听、解码、解析，导致应用系统编码冗余、数据交叉难维护，数据易丢失，资源消耗大。
发明内容
本申请实施例的第一方面提供一种数据处理方法。
本申请实施例第二方面提供一种数据处理装置。
本申请实施例第三方面提供一种服务器。
本申请实施例第四方面提供一种计算机可读存储介质。
根据本申请实施例第一方面的技术方案，提供了一种数据处理方法，包括：为同一类型的联网设备配置对应的通信端口，以使同一类型的多个联网设备通过通信端口传输消息数据，其中，若存在多种类型的联网设备，则为每一种类型的联网设备均配置相应的通信端口；通过至少一个消息中间件将至少一个通信端口监听到的消息数据转接至相应的编码服务，由编码服务对消息数据进行解码，生成解码数据，其中，将消息数据转接至消息数据相应的编码服务之前，对消息数据进行编码格式扩展。
根据本申请实施例第二方面的技术方案，提供了一种数据处理装置，包括： 存储器、处理器及存储在存储器上并可在处理器上运行的程序，程序被处理器执行时实现如上述任一项技术方案的数据处理方法。
根据本申请实施例第三方面的技术方案，还提供了一种服务器，包括：上述第二方面技术方案所述的数据处理装置。
根据本申请实施例第四方面的技术方案，还提供了一种计算机可读存储介质，其上存储有计算机程序，上述计算机程序被执行时实现上述第一方面的任一项技术方案限定的数据处理方法。
附图说明
图1示出了根据本申请的一个实施例的数据处理方法的示意流程图；
图2示出了根据本申请的另一个实施例的数据处理方法的示意图；
图3示出了根据本申请的一个实施例的数据处理装置的示意框图；
图4示出了根据本申请的一个实施例的服务器的示意框图；
图5示出了根据本申请的一个实施例的计算机可读存储介质的示意框图。
具体实施方式
下面结合附图和实施例对本申请进行描述。
在下面的描述中阐述了很多细节以便于理解本申请，但是，本申请还可以采用其他不同于在此描述的其他方式来实施，因此，本申请的保护范围并不受下面公开的实施例的限制。
实施例一
如图1所示，根据本申请的一个实施例的数据处理方法，包括：
步骤102，为同一类型的联网设备配置对应的通信端口，以使同一类型的多个联网设备通过通信端口传输消息数据。
步骤104，通过至少一个消息中间件将至少一个通信端口监听到的消息数据转接至消息数据相应的编码服务，由编码服务对消息数据进行解码，生成解码数据。
步骤106，根据每个联网设备的数据协议调用多线程池服务，将所述每个联网设备的解码数据分配到多线程池服务中的执行线程中进行解析，生成解析数据。
在该实施例中，同时面对多种类型的联网设备，本申请提出为同一类型的多个联网设备配置一个统一的通信端口(分布式数据监听)，同一类型的联网设备连接该统一通信端口进行通信，与其它类型联网设备上传的消息进行逻辑分离，保证多种类型联网设备的消息的原子性。其中，对于多种类型的联网设备都对应设置独立的通信端口即存在多种类型的联网设备的环境中，为每一种类型的联网设备均配置相应的通信端口，消息中间件用于处理多种类型的多个联网设备的消息数据。对于多个通信端口可设置一个消息中间件进行数据处理，也可以设置多个消息中间件进行数据处理，通过消息中间件对通信端口监听到的消息数据进行中转分发，将同一类编码格式的消息数据转接至相应的编码服务进行统一解码，提高解码效率，同时也方便对消息数据的编码格式进行扩展。解码后的消息数据会根据联网设备要求的数据协议调用多线程池服务，通过多线程池服务执行解析过程，多线程池服务的线程池管理技术支持让多个数据流进行并行流转、解析，解析完成的消息数据就是实际业务含义的设备指标数据，也就实现了从物联数据到直观互联数据的转接。
根据上述技术方案，分布式数据监听服务、消息解码服务和多线程异步解析服务能够抽象为API供系统调用，理论上支持的联网设备的类型和数量没有上限。此外，对于消息数据的编码格式、数据协议、数据解析字段及类型能够进行可配置接入，从而实现对联网设备多类型、多设备、多编码、多协议高并发的兼容。
在上述实施例中，可选地，为同一类型的联网设备配置对应的通信端口，以使同一类型的多个联网设备通过通信端口传输消息数据，包括：为同一类型的联网设备配置统一的传输控制协议/网际协议(Transmission Control Protocol/Internet Protocol，TCP/IP)端口，以使同一类型的多个联网设备通过TCP/IP端口传输消息数据，其中，TCP/IP端口用于实现物联网和互联网对接。
在该实施例中，对同一个类型的设备配置一个独立的TCP/IP端口，同一类型的设备通过连接指定端口实现和其他类型设备上传的消息进行逻辑分离，从而保证多种类型设备消息的原子性。同时通过TCP/IP端口的tcp/ip协议实现物联网和互联网对接。基于socket的连接方式，保证数据消息传输的可靠性。基于socket的连接方式，保证数据消息传输的可靠性。其中，socket是基于BSD UNIX的进程通信机制，用于描述IP地址和端口，称为“套接字”，是一个通信句柄，用于实现不同虚拟机或计算设备之间的通信。在网络上主机(server端)一般运行多个应用，并提供多种服务，每种服务都对应一个独立的端口。客户软件通过socket连接到不同的端口，实现数据交互。TCP/IP 称为传输控制协议/互联网协议，其中网络地址统一分配，网络中每一设备和终端都具有一个唯一地址。
在上述任一实施例中，可选地，通过至少一个消息中间件将通信端口监听到的消息数据转接至消息数据相应的编码服务，包括：通信端口监听到消息数据后，通过至少一个基于异步传输原理进行数据消息并行接收和发布的消息中间件，将消息数据转接至消息数据相应的编码服务。
在上述任一实施例中，可选地，通过至少一个消息中间件将通信端口监听到的消息数据转接至消息数据相应的编码服务，由编码服务对消息数据进行解码，生成解码数据，包括：通信端口监听到消息数据后，通过kafka消息队列将消息数据转接至消息数据相应的编码服务，由编码服务对消息数据进行解码，生成解码数据。
在该实施例中，kafka是一种高吞吐量的分布式发布订阅消息中间件，具有高性能、持久化、多副本备份、横向扩展能力。通过kafka消息队列对每个端口接收(监听)到的消息数据进行中转，提高消息数据传输效率。kafka消息队列将具有相同编码格式的消息数据转接至相应的编码服务进行解码，提升对编码格式的兼容度，提高了解码效率。kafka消息队列能够实时处理高并发的大量数据，延迟低，容错性能好。
在上述任一实施例中，可选地，通过消息中间件将通信端口监听到的消息数据转接至消息数据相应的编码服务，由编码服务对消息数据进行解码，生成解码数据，其中，将消息数据转接至消息数据相应的编码服务之前，对消息数据进行编码格式扩展，包括：通过至少一个消息中间件对至少一个所述通信端口监听到的消息数据进行并行缓存，根据联网设备的类型将对应的消息数据入队至带端口号的消息主题中；消息数据入队成功后断开数据传输的长连接；消息数据出队时，对消息数据的编码格式进行扩展，并根据扩展后的消息数据的编码格式对所述消息数据进行分组；根据分组后的消息数据调用相应的编码服务对消息数据进行解码，以生成解码数据，其中，端口号与通信端口一一对应。
在上述任一项实施例中，可选地，通过消息中间件将通信端口监听到的消息数据转接至相应的编码服务，由编码服务对消息数据进行解码，生成解码数据，包括：利用kafka消息队列对消息数据进行并行缓存，根据联网设备的类型将对应的消息数据入队至带端口号的消息主题中；消息数据入队成功后断开数据传输的长连接；消息数据出队时，根据消息主题的端口号调用MINA框架对消息数据的编码格式进行扩展，并对消息数据进行组装分类；根据分类调用相应的编码服务对消息数据进行解码，以生成解码数据，其中，端口号与通信端口一一对应。
本实施例中，对消息数据进行组装分类与对消息数据进行分组的含义相同。例如，可将编码格式相同的消息数据分到同一组，进而根据该组消息数据的编码格式调用响应的编码服务对该组消息数据进行解码，得到解码数据。
在该实施例中，面对多个通信端口监听到的大量消息数据，kafka消息队列对消息数据进行并行缓存，消息数据入队至相应的消息主题，消息主题与通信端口的端口号建立联系，消息数据入队后断开当前的tcp连接，以免长连接增加系统开销。消息数据出队时，根据端口号调用MINA框架对该消息主题中的消息数据进行编码格式的扩展，以便实现物联网和互联网编码无缝衔接。例如，传统物联网编码格式只支持美国信息交换标准代码(American Standard Code for Information Interchange，ASCII)和统一码(UNICODE)，通过MINA框架可以对消息数据的编码格式进行扩展以便解码更多的数据类型。
其中，MINA是一种源于Java nio技术基于TCP/IP和UDP/IP协议提供抽象的、事件驱动的、异步的API。可扩展的编码格式包括：汉字内码扩展规范(Chinese Internal Code Specification，GBK)、8位统一码转换格式(8-bit Unicode Transformation Format，UTF-8)、Unicode、信息交换通用符号编码(Universal Semiotic Coding for Information Interchange，USCII)和16位统一码转换格式(16-bit Unicode Transformation Format，UTF-16)。
在上述任一项实施例中，可选地，还包括：检测到解码过程出现错误，利用MINA框架调用回写接口将错误码返回解码过程出现错误的消息数据所属的联网设备，或者，检测到解码过程出现错误或提示编码不符合编码规范，请求解码过程出现解码错误或提示编码不符合编码规范的消息数据所属的联网设备进行消息重传。
在该实施例中，利用MINA框架对解码过程出现的错误进行矫正，提高解码准确度，提高消息数据解码过程的可靠性。
在上述任一项实施例中，可选地，根据每个联网设备的数据协议调用多线程池服务，将所述每个联网设备的解码数据分配到多线程池服务中的执行线程中进行解析，生成解析数据，包括：根据每个联网设备要求的数据协议调用Executor线程池框架，将所述每个联网设备的解码数据分配到Executor线程池框架的执行线程中进行解析，生成解析数据，其中，执行线程调用解析接口指向相应的数据协议解析方法。
在该实施例中，利用Executor多线程管理技术实现数据并行解析，保证了数据解析效率和及时性。
在该技术方案中，TCP报文一般包含表头、认证、加解密、报文内容等。由于每一种联网设备类型都有特定数据标准，数据协议在不同的行业中也各不相同。本申请通过抽象接口泛型化，对每一种数据解析通过同一接口进行扩展和实现，并将接口暴露出来供不同类型的联网设备调用。本申请采用Executor多线程管理技术来管理线程，每个线程执行方法中调用解析接口指向相应的解析协议方法。而且executor线程池管理技术支持让多个数据流进行并行流转、解析。解析完后的数据就是实际业务含义的设备指标数据，也就相当于实现了从物联数据到直观互联数据的转接。利用Executor多线程管理技术实现数据并行解析，保证了数据解析效率和及时性。其中，Executor是一种管理线程，以实现线程的重用，降低资源消耗，提高响应速度服务，此服务线程适用于执行异步任务。
在上述任一项实施例中，可选地，还包括：存储、展示或转发解析数据。
在该实施例中，解析数据即经过解析的消息数据，解析出来的数据能够根据实际的业务需要进行数据存储、平台展示、转发第三方应用等等，使得物联网数据能够无缝上传至互联网端加以利用。
在上述任一项实施例中，可选地，还包括：对通信端口的端口监听应用服务进行分布式部署，以实现负载均衡。
在该实施例中，针对同一类型中设备数量较多的情况(例如，超过一百个)，通过将该类型的联网设备所对应的通信端口的监听应用服务进行分布式部署的方式进行负载均衡，保证联网设备数据稳定地成功上传。
实施例二
根据本申请的另一个实施例的数据处理方法，包括：
1、数据监听服务：
对同一个类型的设备配置一个独立的TCP/IP端口，同一类型的设备通过连接同一端口实现和其他类型设备上传的消息进行逻辑分离，从而保证多种类型设备消息的原子性。同时通过TCP/IP协议实现物联网和互联网对接。此外，针对单一设备类型中设备数量较多的情况(超过百个)，系统通过将端口监听应用服务进行分布式部署的方式进行负载均衡，保证物联设备数据稳定地成功上传。
2、消息解码服务：
TCP/IP端口监听到数据消息后，运用kafka对已接收消息数据进行并行缓存，入队成功并断开当前TCP/IP连接以免长连接增加系统开销。不同物联设备类型的消息入队到配置带端口号命名的消息主题中，出队时根据主题上 的端口号调用MINA框架对传统的数据编码格式进行扩展，然后对消息数据进行组装分类，最后根据分类对同类型的编码进行统一解码。如果出现解码错误，MINA框架会调用回写接口将错误码返回设备，提示编码不符合编码规范或者解码错误，要求设备端进行消息重传。
3、数据解析服务：
TCP报文一般包含表头、认证、加解密、报文内容等。由于每一种设备类型都有指定数据标准，数据协议在不同的行业中也各自有不同。方案对此通过抽象接口泛型化，对每一种数据解析通过同一接口进行扩展和实现，并将接口暴露出来供不同类型设备调用。我们采用Executor多线程管理技术来管理线程，每个线程执行过程中调用解析接口指向相应的数据协议解析方法。而且Executor线程池管理技术支持让多个数据流进行并行流转、解析。解析完后的数据就是实际业务含义的设备指标数据，也就相当于实现了从物联数据到直观互联数据的转接。
根据上述数据处理方法，本申请的数据传输流程如图2所示，数据传输流程主要包括：根据物联网设备的设备类型将物联网设备分类，为每一种设备类型(例如，包括：设备类型1、设备类型2、设备类型3、设备类型4……设备类型n)的物联网设备分别配置相应的TCP/IP端口(例如，包括：监听端口1、监听端口2、监听端口3、监听端口4……监听端口n)，TCP/IP端口监听到的消息数据通过kafka消息队列进行入队和出队操作，并由MINA框架进行编码格式扩展和根据编码格式进行分组，根据分组情况将消息数据转接至相应的解码服务(例如，包括：解码方式1、解码方式2和解码方式3)，解码后的消息数据再进行数据解析，由物联网数据转换为互联网数据以进行数据应用。其中，n为大于或等于5的正整数。
本实施例实现一种面向多设备类型、多频率、多协议、多编码的高并发物联网数据传输交互的技术方案。实施例分为分布式端口监听、数据解码和多线程异步解析三个应用子系统服务。分布式监听针对不同类型的设备分别配置独立的tcp端口，所有相同类型的设备走同一端口进行消息传输，并通过kafka转接到不同的编码服务进行消息解码。对解码后的数据根据设备要求的数据协议调用多线程池服务分配到一个执行线程中进行解析，解析出来的数据根据实际的业务需要进行数据存储、平台展示、转发第三方应用等等，失败则通过MINA框架回写提示错误。这三种服务均已抽象为应用程序接口(Application Programming Interface，API)供系统调用，理论上对设备的类型、数量不限，方案提供支持GBK、UTF-8、Unicode、USCII、UTF-16等5种传统物联数据编码格式的解码。此外，该实施例还提供对编码格式、数据 协议、数据解析字段及类型实现可配置接入。从而实现对物联设备多类型、多设备、多编码、多协议高并发的兼容。
实施例三
如图3所示，根据本申请的一个实施例的数据处理装置300，包括：存储器302、处理器304及存储在存储器302上并可在处理器304上运行的程序，程序被处理器304执行时实现如上述任一实施例的数据处理方法。
实施例四
如图4所示，根据本申请的一个实施例的服务器400，包括：实施例三所述的数据处理装置300。该服务器400运行时能够实现：为同一类型的联网设备配置对应的通信端口，以使同一类型的多个联网设备通过通信端口传输消息数据；通过消息中间件将通信端口监听到的消息数据转接至所述消息数据相应的编码服务，由编码服务对消息数据进行解码，生成解码数据；根据每个联网设备的数据协议调用多线程池服务，将所述每个联网设备的解码数据分配到多线程池服务中的执行线程中进行解析，生成解析数据。
实施例五
如图5所示，根据本申请的一个实施例，还提供了一种计算机可读存储介质500，其上存储有计算机程序502，上述计算机程序502被执行时实现上述任一实施例限定的数据处理方法。
在该实施例中，计算机程序502被执行时实现：为同一类型的联网设备配置对应的通信端口，以使同一类型的多个联网设备通过通信端口传输消息数据；通过至少一个消息中间件将至少一个通信端口监听到的消息数据转接至所述消息数据相应的编码服务，由编码服务对消息数据进行解码，生成解码数据；根据每个联网设备的数据协议调用多线程池服务，将所述每个联网设备的解码数据分配到多线程池服务中的执行线程中进行解析，生成解析数据。为同一类型的多个联网设备配置一个统一的通信端口，同一类型的联网设备连接该统一通信端口进行通信，与其它类型联网设备上传的消息进行逻辑分离，保证各类型联网设备的消息的原子性。其中，对于多种类型的联网设备都对应设置独立的通信端口，消息中间件用于处理多种类型的多个联网设备的消息数据。通过消息中间件对通信端口监听到的消息数据进行中转分发，将同一类编码格式的消息数据转接至相应的编码服务进行统一解码，提高解码效率，同时也方便对消息数据的编码格式进行扩展。解码后的消息数据会根据联网设备要求的数据协议调用多线程池服务，通过多线程池服务执行解析过程，多线程池服务的线程池管理技术支持让多个数据流进行并行流 转、解析，解析完成的消息数据就是实际业务含义的设备指标数据，也就实现了从物联数据到直观互联数据的转接。
根据上述技术方案的计算机程序502，可选地，为同一类型的联网设备配置对应的通信端口，以使同一类型的多个联网设备通过通信端口传输消息数据，包括：为同一类型的联网设备配置统一的TCP/IP端口，以使同一类型的多个联网设备通过TCP/IP端口传输消息数据，其中，TCP/IP端口用于实现物联网和互联网对接。
在该技术方案中，对同一个类型的设备配置一个独立的TCP/IP端口，同一类型的设备通过连接指定端口实现和其他类型设备上传的消息进行逻辑分离，从而保证各类型设备消息的原子性。同时通过TCP/IP端口的tcp/ip协议实现物联网和互联网对接。基于socket的连接方式，保证数据消息传输的可靠性。
其中，socket是基于BSD UNIX的进程通信机制，用于描述IP地址和端口，称为“套接字”，是一个通信句柄，用于实现不同虚拟机或计算设备之间的通信。在网络上主机(server端)一般运行多个应用，并提供多种服务，每种服务都对应一个独立的端口。客户软件通过socket连接到不同的端口，实现数据交互。TCP/IP称为传输控制协议/互联网协议，其中网络地址统一分配，网络中每一设备和终端都具有一个唯一地址。
根据上述任一项技术方案的计算机程序502，可选地，通过消息中间件将通信端口监听到的消息数据转接至相应的编码服务，由编码服务对消息数据进行解码，生成解码数据，包括：通信端口监听到消息数据后，通过kafka消息队列将消息数据转接至相应的编码服务，由编码服务对消息数据进行解码，生成解码数据。
在该技术方案中，kafka是一种高吞吐量的分布式发布订阅消息中间件，具有高性能、持久化、多副本备份、横向扩展能力。通过kafka消息队列对每个端口接收(监听)到的消息数据进行中转，提高消息数据传输效率。kafka消息队列将具有相同编码格式的消息数据转接至相应的编码服务进行解码，提升对编码格式的兼容度，提高了解码效率。kafka消息队列能够实时处理高并发的大量数据，延迟低，容错性能好。
根据上述任一项技术方案的计算机程序502，可选地，通过消息中间件将通信端口监听到的消息数据转接至相应的编码服务，由编码服务对消息数据进行解码，生成解码数据，包括：利用kafka消息队列对消息数据进行并行缓存，根据联网设备的类型将对应的消息数据入队至带端口号的消息主题中；消息数据入队成功后断开数据传输的长连接；消息数据出队时，根据消 息主题的端口号调用MINA框架对消息数据的编码格式进行扩展，并对消息数据进行组装分类；根据分类调用相应的编码服务对消息数据进行解码，以生成解码数据，其中，端口号与通信端口一一对应。
在该技术方案中，面对多个通信端口监听到的大量消息数据，kafka消息队列对消息数据进行并行缓存，消息数据入队至相应的消息主题，消息主题与通信端口的端口号建立联系，消息数据入队后断开当前的tcp连接，以免长连接增加系统开销。消息数据出队时，根据端口号调用MINA框架对该消息主题中的消息数据进行编码格式的扩展，以便实现物联网和互联网编码无缝衔接。例如，传统物联网编码格式只支持ASCII和UNICODE，通过MINA框架可以对其进行扩展以便解码更多的数据类型。
根据上述任一项技术方案的计算机程序502，可选地，还包括：检测到解码过程出现错误，利用MINA框架调用回写接口将错误码返回解码过程出现错误的消息数据所属的联网设备，或者，检测到解码过程出现错误或提示编码不符合编码规范，请求解码过程出现解码错误或提示编码不符合编码规范的消息数据所属的联网设备进行消息重传。
在该技术方案中，利用MINA框架对解码过程出现的错误进行矫正，提高解码准确度，提高消息数据解码过程的可靠性。
根据上述任一项技术方案的计算机程序502，可选地，根据每个联网设备的数据协议调用多线程池服务，将所述每个联网设备的解码数据分配到多线程池服务中的执行线程中进行解析，生成解析数据，包括：根据每个联网设备要求的数据协议调用Executor线程池框架，将所述每个联网设备的解码数据分配到Executor线程池框架的执行线程中进行解析，生成解析数据，其中，执行线程调用解析接口指向相应的数据协议解析方法。
在该技术方案中，利用Executor多线程管理技术实现数据并行解析，保证了数据解析效率和及时性。
根据上述任一项技术方案的计算机程序502，可选地，还包括：存储、展示或转发解析数据。
在该技术方案中，解析数据即经过解析的消息数据，解析出来的数据能够根据实际的业务需要进行数据存储、平台展示、转发第三方应用等等，使得物联网数据能够无缝上传至互联网端加以利用。
根据上述任一项技术方案的计算机程序502，可选地，还包括：对通信端口的端口监听应用服务进行分布式部署，以实现负载均衡。
在该技术方案中，针对同一类型中设备数量较多的情况(例如，超过一 百个)，通过将该类型的联网设备所对应的通信端口的监听应用服务进行分布式部署的方式进行负载均衡，保证联网设备数据稳定地成功上传。
根据上述实施例，本申请对一种设备类型配置一个独立端口，根据端口分配资源进行入队、解码、解析，实现不同类型数据互不干扰。运用了MINA框架可以对传统的编码进行扩展。从而实现物联到互联编码的无缝衔接。kafka实现消息入队即断开TCP连接，并用Executor进行线程池管理来进行并发的数据处理，提高服务器资源的利用率。
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、光盘只读存储器(Compact Disc-Read Only Memory，CD-ROM)、光学存储器等)上实施的计算机程序产品的形式。
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
应当注意的是，在权利要求中，不应将位于括号之间的任何参考符号构造成对权利要求的限制。单词“包含”不排除存在未列在权利要求中的部件或步骤。位于部件之前的单词“一”或“一个”不排除存在多个这样的部件。本申请可以借助于包括有若干不同部件的硬件以及借助于适当编程的计算机来实现。在列举了若干装置的单元权利要求中，这些装置中的若干个可以是通过 同一个硬件项来体现。单词第一、第二、以及第三等的使用不表示任何顺序。可将这些单词解释为名称。

Claims (11)

  1. 一种数据处理方法，包括：
    为同一类型的联网设备配置对应的通信端口，以使同一类型的多个所述联网设备通过所述通信端口传输消息数据，其中，在存在多种类型的联网设备的情况下，为每一种类型的联网设备配置相应的通信端口；
    通过至少一个消息中间件将至少一个所述通信端口监听到的所述消息数据转接至所述消息数据相应的编码服务，由所述编码服务对所述消息数据进行解码，生成解码数据，其中，将所述消息数据转接至所述消息数据相应的编码服务之前，对所述消息数据进行编码格式扩展。
  2. 根据权利要求1所述的数据处理方法，还包括：
    根据每个联网设备的数据协议调用多线程池服务，将所述每个联网设备的解码数据分配到所述多线程池服务中的执行线程中进行解析，生成解析数据。
  3. 根据权利要求1所述的数据处理方法，其中，所述为同一类型的联网设备配置对应的通信端口，以使同一类型的多个所述联网设备通过所述通信端口传输消息数据，包括：
    为同一类型的联网设备配置统一的传输控制协议/互联网协议TCP/IP端口，以使同一类型的多个所述联网设备通过所述TCP/IP端口传输所述消息数据，其中，所述TCP/IP端口用于实现物联网和互联网对接。
  4. 根据权利要求1所述的数据处理方法，其中，所述通过至少一个消息中间件将至少一个所述通信端口监听到的所述消息数据转接至所述消息数据相应的编码服务，包括：
    在至少一个所述通信端口监听到所述消息数据后，通过至少一个基于异步传输原理进行数据消息并行接收和发布的消息中间件，将所述消息数据转接至所述消息数据相应的编码服务。
  5. 根据权利要求1所述的数据处理方法，其中，所述通过至少一个消息中间件将至少一个所述通信端口监听到的所述消息数据转接至所述消息数据相应的编码服务，由所述编码服务对所述消息数据进行解码，生成解码数据，其中，将所述消息数据转接至所述消息数据相应的编码服务之前，对所述消息数据进行编码格式扩展，包括：
    通过至少一个所述消息中间件对至少一个所述通信端口监听到的所述消息数据进行并行缓存；
    根据所述联网设备的类型将对应的消息数据入队至带端口号的消息主题中；
    在所述消息数据入队成功后，断开数据传输的长连接；
    在所述消息数据出队时，对所述消息数据的编码格式进行扩展，并根据扩展后的所述消息数据的编码格式对所述消息数据进行分组；
    根据分组后的消息数据调用相应的编码服务对所述消息数据进行解码，以生成所述解码数据，
    其中，所述端口号与所述通信端口一一对应。
  6. 根据权利要求5所述的数据处理方法，还包括：
    检测到解码过程出现错误，调用回写接口将错误码返回所述解码过程出现错误的消息数据所属的联网设备，或者，
    检测到解码过程出现错误或提示编码不符合编码规范，请求所述解码过程出现解码错误或提示编码不符合编码规范的消息数据所属的联网设备进行消息重传。
  7. 根据权利要求2所述的数据处理方法，还包括：
    存储、展示或转发所述解析数据。
  8. 根据权利要求1至5中任一项所述的数据处理方法，还包括：
    对所述通信端口的端口监听应用服务进行分布式部署，以实现负载均衡。
  9. 一种数据处理装置，包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的程序，所述程序被所述处理器执行时能够实现如权利要求1至8中任一项所述的数据处理方法。
  10. 一种服务器，包括：
    如权利要求9所述的数据处理装置。
  11. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被执行时，实现如权利要求1至8中任一项所述的数据处理方法。
PCT/CN2020/098781 2019-09-02 2020-06-29 数据处理方法、装置、服务器和计算机可读存储介质 WO2021042840A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910846061.2A CN110677277B (zh) 2019-09-02 2019-09-02 数据处理方法、装置、服务器和计算机可读存储介质
CN201910846061.2 2019-09-02

Publications (1)

Publication Number Publication Date
WO2021042840A1 true WO2021042840A1 (zh) 2021-03-11

Family

ID=69076625

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/098781 WO2021042840A1 (zh) 2019-09-02 2020-06-29 数据处理方法、装置、服务器和计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN110677277B (zh)
WO (1) WO2021042840A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113691553A (zh) * 2021-08-31 2021-11-23 正元地理信息集团股份有限公司 一种市政管网物联网统一接入的方法
CN115665218A (zh) * 2022-12-28 2023-01-31 南方电网数字电网研究院有限公司 一种物联网设备远程控制方法、系统及相关设备
CN116028247A (zh) * 2023-03-27 2023-04-28 中新宽维传媒科技有限公司 一种基于消息中间件的文件采集系统
CN117522601B (zh) * 2024-01-04 2024-03-22 山东保医通信息科技有限公司 一种基于接口统一管理的核保业务执行方法、设备及介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110677277B (zh) * 2019-09-02 2021-12-28 深圳中兴网信科技有限公司 数据处理方法、装置、服务器和计算机可读存储介质
CN113904951B (zh) * 2021-09-28 2023-07-25 济南浪潮数据技术有限公司 一种网关设备的数据监控方法、装置及相关设备
CN113821361B (zh) * 2021-11-24 2022-05-17 中节能晶和智慧城市科技(浙江)有限公司 一种基于流式处理的物联网平台消息处理方法和系统
CN114567416A (zh) * 2022-04-06 2022-05-31 西安羚控电子科技有限公司 基于tcp数据传输免校验方法及系统
CN116112536A (zh) * 2022-12-30 2023-05-12 广州通康创智软件有限公司 一种tcp-串口通信透传的方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101719927A (zh) * 2009-11-18 2010-06-02 中兴通讯股份有限公司 对网关下挂的设备实行远程管理的方法及其系统
CN102546461A (zh) * 2011-12-13 2012-07-04 中国电子科技集团公司第十五研究所 多层级消息中间件系统及其消息转发控制方法和装置
US20170054770A1 (en) * 2015-08-23 2017-02-23 Tornaditech Llc Multimedia teleconference streaming architecture between heterogeneous computer systems
US20180359259A1 (en) * 2017-06-09 2018-12-13 ORock Holdings, LLC Secure network-accessible system for executing remote applications
CN109933575A (zh) * 2019-02-28 2019-06-25 鲁东大学 监测数据的存储方法及装置
CN110677277A (zh) * 2019-09-02 2020-01-10 深圳中兴网信科技有限公司 数据处理方法、装置、服务器和计算机可读存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101050282B1 (ko) * 2010-10-29 2011-07-19 한화에스앤씨주식회사 멀티 프로토콜 어댑터 시스템 및 멀티 프로토콜 어댑터 시스템에서의 데이터 변환 방법
CN102045896A (zh) * 2010-11-22 2011-05-04 中山爱科数字科技有限公司 实现多协议及网络自适应的虚拟物联网网关系统
CN103699692A (zh) * 2014-01-11 2014-04-02 樊建 物联网接入平台数据管理方法
KR101938312B1 (ko) * 2016-10-28 2019-01-14 국민대학교산학협력단 사물인터넷 기반의 DUSS(Different Units Same Security) 장치
CN106534107B (zh) * 2016-11-04 2020-05-12 北方工业大学 一种物联网消息服务系统
CN106878459B (zh) * 2017-03-24 2020-04-03 深圳安邦科技有限公司 一种自适应的物联网智能网关实现方法及其设备
CN107317726A (zh) * 2017-06-21 2017-11-03 深圳市盛路物联通讯技术有限公司 一种物联网数据的上报控制方法及设备
CN108459915A (zh) * 2018-01-31 2018-08-28 中电福富信息科技有限公司 一种适用于高并发场景的tcp通道管理系统及其方法
CN109040144B (zh) * 2018-10-19 2021-09-24 京信网络系统股份有限公司 物联网网关数据处理方法、装置、存储介质及网关设备
CN110109996A (zh) * 2019-05-15 2019-08-09 浪潮通用软件有限公司 一种工业互联网设备实时数据采集与存储方法及系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101719927A (zh) * 2009-11-18 2010-06-02 中兴通讯股份有限公司 对网关下挂的设备实行远程管理的方法及其系统
CN102546461A (zh) * 2011-12-13 2012-07-04 中国电子科技集团公司第十五研究所 多层级消息中间件系统及其消息转发控制方法和装置
US20170054770A1 (en) * 2015-08-23 2017-02-23 Tornaditech Llc Multimedia teleconference streaming architecture between heterogeneous computer systems
US20180359259A1 (en) * 2017-06-09 2018-12-13 ORock Holdings, LLC Secure network-accessible system for executing remote applications
CN109933575A (zh) * 2019-02-28 2019-06-25 鲁东大学 监测数据的存储方法及装置
CN110677277A (zh) * 2019-09-02 2020-01-10 深圳中兴网信科技有限公司 数据处理方法、装置、服务器和计算机可读存储介质

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
"3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Study on Enhancements to the Service-Based Architecture (Release 16)", 3GPP STANDARD; TECHNICAL REPORT; 3GPP TR 23.742, 3RD GENERATION PARTNERSHIP PROJECT (3GPP), MOBILE COMPETENCE CENTRE ; 650, ROUTE DES LUCIOLES ; F-06921 SOPHIA-ANTIPOLIS CEDEX ; FRANCE, no. V0.3.0, 23 July 2018 (2018-07-23), Mobile Competence Centre ; 650, route des Lucioles ; F-06921 Sophia-Antipolis Cedex ; France, pages 1 - 65, XP051475058 *
RAUSCH THOMAS; DUSTDAR SCHAHRAM; RANJAN RAJIV: "Osmotic Message-Oriented Middleware for the Internet of Things", IEEE CLOUD COMPUTING, IEEE, USA, vol. 5, no. 2, 1 March 2018 (2018-03-01), USA, pages 17 - 25, XP011683036, DOI: 10.1109/MCC.2018.022171663 *
WEI, YI: "The Design and Implementation of the Message-oriented Middleware MTQ Based on MINA", MASTER'S THESES , DALIAN UNIVERSITY OF TECHNOLOGY, 22 May 2016 (2016-05-22), pages 1 - 63, XP055788273, [retrieved on 20210322] *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113691553A (zh) * 2021-08-31 2021-11-23 正元地理信息集团股份有限公司 一种市政管网物联网统一接入的方法
CN113691553B (zh) * 2021-08-31 2022-12-20 正元地理信息集团股份有限公司 一种市政管网物联网统一接入的方法、终端、系统和存储介质
CN115665218A (zh) * 2022-12-28 2023-01-31 南方电网数字电网研究院有限公司 一种物联网设备远程控制方法、系统及相关设备
CN115665218B (zh) * 2022-12-28 2023-04-25 南方电网数字电网研究院有限公司 一种物联网设备远程控制方法、系统及相关设备
CN116028247A (zh) * 2023-03-27 2023-04-28 中新宽维传媒科技有限公司 一种基于消息中间件的文件采集系统
CN117522601B (zh) * 2024-01-04 2024-03-22 山东保医通信息科技有限公司 一种基于接口统一管理的核保业务执行方法、设备及介质

Also Published As

Publication number Publication date
CN110677277A (zh) 2020-01-10
CN110677277B (zh) 2021-12-28

Similar Documents

Publication Publication Date Title
WO2021042840A1 (zh) 数据处理方法、装置、服务器和计算机可读存储介质
CN108647104B (zh) 请求处理方法、服务器及计算机可读存储介质
US7274706B1 (en) Methods and systems for processing network data
EP0899659B1 (en) An apparatus and method for simulating multiple nodes on a single machine
TWI479850B (zh) 單數據機板之改良式多胞元支援方法及系統
US11860796B2 (en) Execution space agnostic device drivers
CN111966446B (zh) 一种容器环境下rdma虚拟化方法
US20100064286A1 (en) Data affinity based scheme for mapping connections to cpus in i/o adapter
US10609125B2 (en) Method and system for transmitting communication data
US8280197B1 (en) Managed background loading of image resources in resource-constrained devices
WO2023046141A1 (zh) 一种数据库网络负载性能的加速框架、加速方法及设备
US8135851B2 (en) Object request broker for accelerating object-oriented communications and method
US10313450B2 (en) Method for transparently connecting augmented network socket operations
CN103338156B (zh) 一种基于线程池的命名管道服务器并发通信方法
US11409569B2 (en) Data processing system
WO2021238259A1 (zh) 一种数据传输方法、装置、设备及计算机可读存储介质
US20230153153A1 (en) Task processing method and apparatus
CN113778714B (zh) 一种Linux兼容Android的共享剪切板方法及装置
CN113849449A (zh) 一种通信系统和信息交互方法、设备和介质
JPH06301655A (ja) 分散処理システム
CN114371935A (zh) 网关处理方法、网关、设备及介质
EP1548591B1 (en) Method, apparatuses and instruction set for an accelerator for object-oriented communications
US7320044B1 (en) System, method, and computer program product for interrupt scheduling in processing communication
Miura et al. Xmcapi: Inter-core communication interface on multi-chip embedded systems
Rosa et al. INSANE: A Unified Middleware for QoS-aware Network Acceleration in Edge Cloud Computing

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20860282

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20860282

Country of ref document: EP

Kind code of ref document: A1