CN110912947A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN110912947A CN110912947A CN201811072288.8A CN201811072288A CN110912947A CN 110912947 A CN110912947 A CN 110912947A CN 201811072288 A CN201811072288 A CN 201811072288A CN 110912947 A CN110912947 A CN 110912947A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- sequence
- gateway
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/40—Bus networks
- H04L12/40006—Architecture of a communication node
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/28—Data switching networks characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks]
- H04L12/40—Bus networks
- H04L2012/40208—Bus networks characterized by the use of a particular bus standard
- H04L2012/40228—Modbus
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present disclosure provides a data processing method and apparatus. The data processing technology is related to, and the problem of low data processing efficiency can be solved. The specific technical scheme is as follows: a thread pool is arranged in the acquisition server, the thread pool comprises a plurality of threads running in parallel, and each thread executes the following operations: acquiring a data sequence to be processed from a corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from the terminal equipment which establishes communication connection with the gateway; processing the data sequence to be processed according to a preset processing rule to generate corresponding equipment data; the device data comprises data information, device information and parameter information; and sending the device data to a message queue for acquisition and storage by a processing server. The present disclosure is directed to improving the efficiency of data processing.
Description
Technical Field
The present disclosure relates to the field of data processing, and in particular, to a data processing method and apparatus.
Background
Modbus is a serial communication protocol applied between industrial electronic devices, and is generally used for connecting a server and a terminal device, and allowing the server and a plurality of terminal devices to be connected on the same network for communication. The Modbus communication protocol is a master/slave architecture protocol, wherein one node, such as a server, is a master node, and the other nodes participating in communication, such as terminal devices, are slave nodes.
In the prior art, a gateway acquires binary data from a terminal device, a server acquires the binary data of the terminal device from the gateway in a Modbus TCP connection mode, converts the acquired binary data into actual data, performs corresponding association operation on the actual data and the terminal device, and then sends a data message to a message queue for forwarding and storing.
However, the next acquisition task is started after the data acquired by each acquisition task is converted, associated, forwarded and stored. The single-thread mode occupies more server resources and has lower data processing efficiency.
Disclosure of Invention
The embodiment of the disclosure provides a data processing method and device, which can solve the problem of low data acquisition efficiency. The technical scheme is as follows:
according to a first aspect of embodiments of the present disclosure, there is provided a data processing method, the method including:
a thread pool is set in the acquisition server, the thread pool comprises a plurality of threads running in parallel, wherein each thread executes the following operations:
acquiring a data sequence to be processed from a corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from a terminal device establishing communication connection with the gateway;
processing the data sequence to be processed according to a preset processing rule to generate corresponding equipment data; the device data comprises data information, device information and parameter information;
and sending the equipment data to a message queue for a processing server to obtain and store.
The data processing in the multithreading mode is realized by arranging a plurality of threads running in parallel in the acquisition server, and new data acquisition and processing tasks can be entered after the equipment data are sent to the message queue. The data processing efficiency is improved while server resources are saved.
In an embodiment, the processing the to-be-processed data sequence according to a preset processing rule to generate corresponding device data includes:
performing association operation on the data to be processed after data system conversion and the terminal equipment according to a pre-configured point table and a sequence order to generate corresponding equipment data; and the point table comprises a corresponding terminal equipment sequence and a corresponding parameter sequence.
The point table comprising the corresponding terminal equipment sequence and the parameter sequence is configured in advance, the data sequence to be processed is simply matched and associated with the point table, equipment data are generated, and the data processing efficiency is further improved.
In an embodiment, the obtaining a to-be-processed data sequence from a corresponding gateway includes:
and acquiring the data sequence to be processed from the corresponding gateway through a Modbus TCP communication protocol.
In one embodiment, the device information includes a device type and a device identification.
In one embodiment, the parameter information includes a telemetry parameter and an electrical energy parameter.
According to a second aspect of the embodiments of the present disclosure, there is provided a data processing apparatus, the apparatus comprising:
the acquisition module is used for acquiring the data sequence to be processed from the corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from a terminal device establishing communication connection with the gateway;
the processing module is used for processing the data sequence to be processed according to a preset processing rule to generate corresponding equipment data; the device data comprises data information, device information and parameter information;
and the sending module is used for sending the equipment data to a message queue for a processing server to obtain and store.
After the sending module of the data processing device sends the equipment data to the message queue, the new data acquisition and processing task can be entered, so that the server resources are saved, and the data processing efficiency is improved.
In one embodiment, the processing module is specifically configured to:
performing association operation on the data to be processed after data system conversion and the terminal equipment according to a pre-configured point table and a sequence order to generate corresponding equipment data; and the point table comprises a corresponding terminal equipment sequence and a corresponding parameter sequence.
The processing module is simply matched and associated with the data sequence to be processed through the point table which is configured in advance and comprises the corresponding terminal equipment sequence and the corresponding parameter sequence, equipment data are generated, and the data processing efficiency is further improved.
In one embodiment, the obtaining module is specifically configured to:
and acquiring the data sequence to be processed from the corresponding gateway through a Modbus TCP communication protocol.
In one embodiment, the device information includes a device type and a device identification.
In one embodiment, the parameter information includes a telemetry parameter and an electrical energy parameter.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a flowchart of a data processing method provided by an embodiment of the present disclosure;
fig. 2 is a schematic logical layer structure diagram of a data processing apparatus according to an embodiment of the present disclosure;
FIG. 3 is an interaction diagram of a data processing method provided by an embodiment of the present disclosure;
fig. 4 is a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
The embodiment of the present disclosure provides a data processing method, which is mainly applied to an acquisition server, in the embodiment of the present disclosure, a thread pool is set in the acquisition server, the thread pool includes a plurality of threads running in parallel, and as shown in fig. 1, an operation executed by each thread includes the following steps:
101. acquiring a data sequence to be processed from a corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from the terminal equipment which establishes communication connection with the gateway.
Before the step, the gateway needs to be registered on an acquisition server, and the acquisition server allocates a corresponding thread for the gateway. And the thread acquires the data sequence to be processed from the corresponding gateway through a Modbus TCP communication protocol.
102. Processing the data sequence to be processed according to a preset processing rule to generate corresponding equipment data; the device data includes data information, device information, and parameter information.
It should be noted that, the to-be-processed data in the to-be-processed data sequence acquired by the thread from the gateway is usually binary data, in this step, data system conversion needs to be performed on the to-be-processed data first, the binary data is converted into decimal data, and then the to-be-processed data after data system conversion and the terminal device are subjected to association operation according to a pre-configured point table and sequence order to generate corresponding device data; and the point table comprises a corresponding terminal equipment sequence and a corresponding parameter sequence.
Specifically, the device information includes a device type and a device identifier, for example, the high-voltage switch cabinet 01, the high-voltage switch cabinet 02, and the low-voltage incoming line cabinet 03. The parameter information comprises a remote measurement parameter and an electric energy parameter. The remote measurement parameters may be, for example, voltage, current, active, reactive, power factor, frequency, and the like, and the power parameters may include, for example, forward active power, forward reactive power, and the like.
103. And sending the equipment data to a message queue for a processing server to obtain and store.
In this step, after the thread sends the device data generated in the above step to the message queue, the thread may return to execute the above step 101. The processing server may then retrieve the device data from the message queue for storage.
According to the data processing method provided by the embodiment of the disclosure, the multithreading mode of data processing is realized by arranging a plurality of threads running in parallel in the acquisition server, and after the device data are sent to the message queue, the device data can enter a new data acquisition processing task, the acquisition server and the processing server are not interfered with each other and are executed asynchronously, so that the data processing efficiency is improved, and meanwhile, the server resources are saved.
Based on the data processing method provided by the embodiment corresponding to fig. 1, another embodiment of the present disclosure provides a data processing method. Referring to fig. 2 and fig. 3, the data processing method provided in this embodiment includes the following steps:
301. the gateway collects data to be processed from the terminal equipment and stores the data in the form of a data sequence to be processed.
Specifically, the terminal device may include a primary power device, such as a power meter, a protection device, an environmental information device, and the like.
The gateway can establish communication connection with the terminal equipment through communication protocols such as modbus, 103 and the like, and obtain the data sequence to be processed from the terminal equipment at a preset time interval.
302. The acquisition server establishes connection with the gateway through a network, and registers the gateway on the acquisition server.
The network may be, for example, the internet, a 3G network, a 4G network, etc., and this embodiment is not particularly limited thereto.
303. And the acquisition server acquires the data sequence to be processed from the gateway.
The acquisition server can acquire the data sequence to be processed from the gateway through a modbus TCP communication protocol. Specific techniques that may be applied include, for example, a boost library, a muduo web communication library, etc.
304. And the acquisition server processes the data sequence to be processed according to a preset processing rule, generates corresponding equipment data and sends the equipment data to the rabbitmq message queue.
In this step, the acquisition server performs association operation on the data to be processed and the terminal device according to the sequence order and the parameter sequence in the pre-configured point table, so as to generate device data. The device data includes data information, device information, and parameter information.
It should be noted that rabbitmq is a kind of message middleware implementing AMQP (advanced message queuing protocol), originally originated from a financial system, used for storing and forwarding messages in a distributed system, and is not popular in terms of usability, extensibility, high availability, and the like.
305. And the processing server acquires the device data from the rabbitmq message queue for storage.
According to the data processing method provided by the embodiment of the disclosure, the acquisition server can enter a new data acquisition processing task after sending the equipment data to the message queue, the acquisition server and the processing server are not interfered with each other and are asynchronously executed, so that the data processing efficiency is improved, and meanwhile, the server resources are saved.
Based on the data methods described in the embodiments corresponding to fig. 1 to 3, the following is an embodiment of the apparatus of the present disclosure, which can be used to execute the embodiments of the method of the present disclosure.
The embodiment of the present disclosure provides a data processing apparatus, as shown in fig. 4, the data processing apparatus 40 includes: an acquisition module 401, a processing module 402 and a sending module 403.
The obtaining module 401 is configured to obtain a data sequence to be processed from a corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from the terminal equipment which establishes communication connection with the gateway.
The processing module 402 is configured to process the to-be-processed data sequence according to a preset processing rule, and generate corresponding device data; the device data includes data information, device information, and parameter information. The device information comprises a device type and a device identifier; the parameter information comprises a remote measurement parameter and an electric energy parameter.
The sending module 403 is configured to send the device data to a message queue for the processing server to obtain and store.
Specifically, the processing module is specifically configured to: performing association operation on the data to be processed after data system conversion and the terminal equipment according to a pre-configured point table and a sequence order to generate corresponding equipment data; and the point table comprises a corresponding terminal equipment sequence and a corresponding parameter sequence.
The acquisition module is specifically configured to: and acquiring the data sequence to be processed from the corresponding gateway through a Modbus TCP communication protocol.
The data processing apparatus provided in the embodiment of the present disclosure may be configured to execute the technical solutions of the method embodiments, and the implementation principles and technical effects thereof are similar and will not be described herein again.
Based on the data processing method described in the embodiments corresponding to fig. 1 to fig. 3, embodiments of the present disclosure further provide a computer-readable storage medium, for example, the non-transitory computer-readable storage medium may be a Read Only Memory (ROM), a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. The storage medium stores computer instructions for executing the data processing method described in the embodiment corresponding to fig. 1 to fig. 3, which is not described herein again.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (10)
1. A method of data processing, the method comprising:
a thread pool is set in the acquisition server, the thread pool comprises a plurality of threads running in parallel, wherein each thread executes the following operations:
acquiring a data sequence to be processed from a corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from a terminal device establishing communication connection with the gateway;
processing the data sequence to be processed according to a preset processing rule to generate corresponding equipment data; the device data comprises data information, device information and parameter information;
and sending the equipment data to a message queue for a processing server to obtain and store.
2. The method according to claim 1, wherein the processing the to-be-processed data sequence according to a preset processing rule to generate corresponding device data comprises:
performing association operation on the data to be processed after data system conversion and the terminal equipment according to a pre-configured point table and a sequence order to generate corresponding equipment data; and the point table comprises a corresponding terminal equipment sequence and a corresponding parameter sequence.
3. The method of claim 1, wherein the obtaining the to-be-processed data sequence from the corresponding gateway comprises:
and acquiring the data sequence to be processed from the corresponding gateway through a Modbus TCP communication protocol.
4. The method of claim 1, wherein the device information comprises a device type and a device identification.
5. The method of claim 1, wherein the parameter information comprises a telemetry parameter and an electrical energy parameter.
6. A data processing apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring the data sequence to be processed from the corresponding gateway; the data sequence to be processed comprises a plurality of data to be processed; the data to be processed is acquired by the gateway from a terminal device establishing communication connection with the gateway;
the processing module is used for processing the data sequence to be processed according to a preset processing rule to generate corresponding equipment data; the device data comprises data information, device information and parameter information;
and the sending module is used for sending the equipment data to a message queue for a processing server to obtain and store.
7. The apparatus of claim 6, wherein the processing module is specifically configured to:
performing association operation on the data to be processed after data system conversion and the terminal equipment according to a pre-configured point table and a sequence order to generate corresponding equipment data; and the point table comprises a corresponding terminal equipment sequence and a corresponding parameter sequence.
8. The apparatus of claim 6, wherein the obtaining module is specifically configured to:
and acquiring the data sequence to be processed from the corresponding gateway through a Modbus TCP communication protocol.
9. The apparatus of claim 6, wherein the device information comprises a device type and a device identifier.
10. The apparatus of claim 6, wherein the parameter information comprises a telemetry parameter and an electrical energy parameter.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811072288.8A CN110912947A (en) | 2018-09-14 | 2018-09-14 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811072288.8A CN110912947A (en) | 2018-09-14 | 2018-09-14 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110912947A true CN110912947A (en) | 2020-03-24 |
Family
ID=69812325
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811072288.8A Pending CN110912947A (en) | 2018-09-14 | 2018-09-14 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110912947A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112671933A (en) * | 2021-02-23 | 2021-04-16 | 浙江中控技术股份有限公司 | Data processing method and system |
| CN113296972A (en) * | 2020-07-20 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Information registration method, computing device and storage medium |
| WO2022166255A1 (en) * | 2021-02-04 | 2022-08-11 | 浙江所托瑞安科技集团有限公司 | Data processing method and apparatus, and server and computer-readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491190A (en) * | 2013-09-30 | 2014-01-01 | 国家电网公司 | Processing method for large-scale real-time concurrent charger monitoring data |
| CN106993058A (en) * | 2017-05-24 | 2017-07-28 | 儒安科技有限公司 | The transfer method and apparatus of network request |
| US20180007173A1 (en) * | 2016-07-01 | 2018-01-04 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Data Processing Method and Apparatus for Performing Protocol Parsing in a Cloud |
-
2018
- 2018-09-14 CN CN201811072288.8A patent/CN110912947A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103491190A (en) * | 2013-09-30 | 2014-01-01 | 国家电网公司 | Processing method for large-scale real-time concurrent charger monitoring data |
| US20180007173A1 (en) * | 2016-07-01 | 2018-01-04 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Data Processing Method and Apparatus for Performing Protocol Parsing in a Cloud |
| CN106993058A (en) * | 2017-05-24 | 2017-07-28 | 儒安科技有限公司 | The transfer method and apparatus of network request |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113296972A (en) * | 2020-07-20 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Information registration method, computing device and storage medium |
| WO2022166255A1 (en) * | 2021-02-04 | 2022-08-11 | 浙江所托瑞安科技集团有限公司 | Data processing method and apparatus, and server and computer-readable storage medium |
| CN112671933A (en) * | 2021-02-23 | 2021-04-16 | 浙江中控技术股份有限公司 | Data processing method and system |
| CN112671933B (en) * | 2021-02-23 | 2022-04-26 | 浙江中控技术股份有限公司 | Data processing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111580995B (en) | Synchronous communication method and system of distributed cloud platform and Internet of things intelligent terminal based on MQTT asynchronous communication scene | |
| CN110912947A (en) | Data processing method and device | |
| CN110839084B (en) | Session management method, device, equipment and medium | |
| CN108712485A (en) | A kind of resource subscription method and apparatus of internet of things equipment | |
| CN113273135B (en) | Network topology discovery in a substation | |
| CN111481921B (en) | Scheduling method, device, equipment and storage medium of cloud game instance | |
| CN111541762B (en) | Data processing method, management server, device and storage medium | |
| CN110536359B (en) | Network slice creation method, device, equipment and storage medium | |
| CN107925652B (en) | Fixed conversion method and device of transmission protocol based on message protocol identification | |
| CN111240858A (en) | Event scheduling method and component | |
| CN108965359B (en) | Communication method, communication device, readable medium, and electronic apparatus | |
| CN114500633B (en) | Data forwarding method, related device, program product and data transmission system | |
| US10478731B2 (en) | Method and apparatus for storing user data | |
| CN112104679B (en) | Method, apparatus, device and medium for processing hypertext transfer protocol request | |
| CN112333284B (en) | 5G intelligent gateway-based data transmission method, system and storage medium | |
| CN103607324B (en) | Data processing method for Java message service, Java message client and server | |
| CN110445848A (en) | Method and apparatus for issued transaction | |
| CN115150420B (en) | Service processing method and device and related equipment | |
| CN110661681B (en) | Buried point design method and device | |
| US20240048644A1 (en) | Message transmission method and system, and network device and storage medium | |
| CN115269161B (en) | Abnormal process reverse rollback method and system | |
| CN113747100B (en) | Audio and video call method and device, storage medium and electronic equipment | |
| CN117707990B (en) | Processing device and method for ground fault information | |
| CN112882853A (en) | Data processing method, device and equipment | |
| CN113810775B (en) | Voice and video connection system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200324 |
|
| RJ01 | Rejection of invention patent application after publication |