CN116193296A - Method, device, equipment and medium for collecting and processing containerized distributed data - Google Patents
Method, device, equipment and medium for collecting and processing containerized distributed data Download PDFInfo
- Publication number
- CN116193296A CN116193296A CN202211699476.XA CN202211699476A CN116193296A CN 116193296 A CN116193296 A CN 116193296A CN 202211699476 A CN202211699476 A CN 202211699476A CN 116193296 A CN116193296 A CN 116193296A
- Authority
- CN
- China
- Prior art keywords
- data
- terminal
- electric energy
- processed
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 63
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000006243 chemical reaction Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 16
- 238000007405 data analysis Methods 0.000 claims description 12
- 238000003672 processing method Methods 0.000 claims description 10
- 239000003245 coal Substances 0.000 claims description 4
- 230000008569 process Effects 0.000 description 13
- 238000004891 communication Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 230000005611 electricity Effects 0.000 description 6
- 238000004519 manufacturing process Methods 0.000 description 6
- 238000007726 management method Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000008014 freezing Effects 0.000 description 3
- 238000007710 freezing Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 206010033799 Paralysis Diseases 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000013497 data interchange Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q2209/00—Arrangements in telecontrol or telemetry systems
- H04Q2209/20—Arrangements in telecontrol or telemetry systems using a distributed architecture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q2209/00—Arrangements in telecontrol or telemetry systems
- H04Q2209/60—Arrangements in telecontrol or telemetry systems for transmitting utility meters data, i.e. transmission of data from the reader of the utility meter
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q2209/00—Arrangements in telecontrol or telemetry systems
- H04Q2209/70—Arrangements in the main station, i.e. central controller
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method, a device, equipment and a medium for collecting and processing containerized distributed data. The method comprises at least one distributed container for performing the steps of: receiving data to be processed acquired by mass terminal equipment; analyzing the data to be processed to obtain data to be stored in a target format; converting the data to be stored according to preset rules to obtain data to be used; and merging the data to be used to obtain target storage data. According to the technical scheme, the containerized distributed nodes are deployed on the computer cluster, so that the to-be-processed data uploaded by mass terminal equipment can be analyzed, converted and combined based on the distributed nodes, the mass data uploaded by different terminal types can be processed in real time, the data processing capacity is enlarged, and meanwhile, the data processing efficiency is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a medium for collecting and processing containerized distributed data.
Background
With the development of the power industry, the metering access ammeter equipment in the power industry reaches the level of hundred million, and the high-frequency acquisition and processing of huge amounts of equipment and massive data are one of the problems to be solved in the current industry.
At present, a processing mode of mass data is to process complex data uploaded by multiple types of terminals simultaneously based on a single server, so that the capacity of processing data is limited by the capacity of the server, the problem of system paralysis easily occurs in the data processing process, the data processing efficiency is low, the processing data capacity is small, and certain time delay exists.
Disclosure of Invention
The invention provides a containerized distributed data acquisition and processing method, device, equipment and medium, which realize real-time processing of mass data uploaded by different terminal types, expand data processing capacity and improve data processing efficiency.
In a first aspect, an embodiment of the present invention provides a method for collecting and processing containerized distributed data, where the method includes:
receiving data to be processed acquired by mass terminal equipment;
analyzing the data to be processed to obtain data to be stored in a target format;
converting the data to be stored according to preset rules to obtain data to be used;
and merging the data to be used to obtain target storage data.
In a second aspect, an embodiment of the present invention further provides a containerized distributed data acquisition processing apparatus, where the apparatus includes:
the data receiving module is used for receiving data to be processed acquired by mass terminal equipment;
the data analysis module is used for analyzing the data to be processed to obtain data to be stored in a target format;
the data conversion module is used for carrying out conversion processing on the data to be stored according to preset rules to obtain data to be used;
and the data merging module is used for merging the data to be used to obtain target storage data.
In a third aspect, the present invention also provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the containerized distributed data acquisition processing method of any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to implement a containerized distributed data acquisition processing method according to any one of the embodiments of the present invention when executed.
According to the technical scheme, the data to be processed acquired by mass terminal equipment are received; analyzing the data to be processed to obtain data to be stored in a target format; converting the data to be stored according to preset rules to obtain data to be used; and merging the data to be used to obtain target storage data. The containerized distributed nodes are deployed on the computer cluster, and the data to be processed uploaded by the mass terminal equipment can be analyzed, converted and combined based on the distributed nodes, so that the mass data uploaded by different terminal types are processed in real time, the data processing capacity is enlarged, and the data processing efficiency is improved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a data processing process provided in accordance with an embodiment of the present invention;
FIG. 2 is a flow chart of a method for processing containerized distributed data acquisition according to a first embodiment of the present invention;
FIG. 3 is a data processing module specification diagram provided in accordance with an embodiment of the present invention;
FIG. 4 is a flow chart of a method for collecting and processing containerized distributed data according to a second embodiment of the present invention;
fig. 5 is a schematic structural diagram of a containerized distributed data acquisition processing device according to a third embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device implementing a containerized distributed data acquisition processing method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Before introducing the technical scheme, an application scenario may be described in an exemplary way: fig. 1 is a schematic diagram of a data processing process according to an embodiment of the present invention, as shown in fig. 1, in a power production process, electric meters configured in various places generate a large amount of production data, the production data are uploaded to corresponding terminal devices, and each terminal device finally uploads the production data to corresponding distributed nodes, where processes such as message analysis, data conversion, data merging and the like are performed on the production data.
Example 1
Fig. 2 is a flowchart of a method for collecting and processing containerized distributed data according to a first embodiment of the present invention, where the method may be implemented by a containerized distributed data collecting and processing device, and the device may be implemented in hardware and/or software, and the device may be configured in a computer device. The computer device may be a notebook, desktop computer, smart tablet, or the like.
As shown in fig. 2, the method includes:
s110, receiving data to be processed acquired by mass terminal equipment.
The terminal equipment is hardware equipment for collecting message data corresponding to the power production data in the ammeter in real time. The communication link can be pre-established between the terminal equipment and the ammeter so as to realize data transmission between the terminal equipment and the ammeter. The number of terminal devices may be massive, and the number of electricity meters to which one terminal device is connected may be one or more, which is not limited in this embodiment. The data to be processed may be electric energy related data including at least one of data of a coal electricity meter, a gas electricity meter, a photovoltaic electricity meter, and the like. Further, the data to be processed may be of the following types: for example, the data to be processed may be message data, freeze class data, curve class data, minute class data, event class data, and the like. The message data mainly comprise heartbeat, control, parameters and the like of the sensing equipment; the freezing type data mainly comprises day/month freezing electric energy indication value, day/month freezing requirement and the like; the curve data mainly comprise electric energy, power, current, voltage, power factor and the like; the minute-level data mainly comprises minute-level electric energy indication values, power and the like; the event type data mainly comprises a power-off event, a metering device abnormal event, a load out-of-limit abnormal event and the like.
It should be noted that, in the embodiment of the present invention, the data to be processed is in hexadecimal format.
Specifically, on each containerized distributed node, the data to be processed uploaded by the corresponding terminal device can be received with a period of 1 second. Further, the number of distributed containerization nodes may be one or more, and one distributed containerization node may receive data to be processed uploaded by one or more terminal devices.
Illustratively, the distributed containerization node 1 receives the data to be processed uploaded by the terminal device a, the terminal device B, and the terminal device C every 1 second.
And S120, analyzing the data to be processed to obtain the data to be stored in the target format.
The parsing process may include the following two aspects: in the first aspect, hexadecimal data to be processed is converted into electric energy related data based on a data analysis module; for some curve types of data to be processed, the data are usually represented by isolated points in the message data, and the points can be combined through a data analysis module so as to obtain complete curve data. The second aspect converts the power-related data obtained by the parsing process into a target format. The data to be stored is the electric energy associated data in the target format. In this embodiment, the target format is json format.
Specifically, since the analysis processing modes of the data to be processed of each containerized distributed node are the same, a description will be given of one containerized distributed node: and at the current containerized distributed node, converting the data to be processed from hexadecimal into the data to be stored in a json format in a target format through a message data analysis module in a program. Since json is a lightweight data interchange format, it stores and represents data in a text format that is completely independent of the programming language. Therefore, the method has the advantages of being easy to read and write by people, easy to analyze and generate by machines, and effectively improving the network transmission efficiency.
S130, converting the data to be stored according to preset rules to obtain data to be used.
The preset rule may be to convert the data collection time in the data to be stored according to the frequency of data collection, or to add a corresponding terminal device identifier for each data in the data to be stored, to run an electric energy meter identifier, etc. The data to be used is data obtained after conversion processing is carried out according to preset rules on the basis of the data to be stored.
Specifically, since the conversion processing modes of the data to be stored of each containerized distributed node are the same, a description will be given of one containerized distributed node: and at the current containerized distributed node, converting the data to be stored by a service data conversion module in the program to obtain the data to be used.
S140, merging the data to be used to obtain target storage data.
The merging process may be to merge all data to be used collected in a certain time range according to different electric meter devices. The target storage data refers to data corresponding to different service data corresponding to each ammeter.
Specifically, since the merging processing manner of the data to be used by each containerized distributed node is the same, a description will be given of one containerized distributed node: at the current containerized distributed node, through a business data merging module in a program, data to be used acquired by the ammeter equipment 1 in a time period of 9:00-9:15 can be merged according to time sequence, and then target storage data corresponding to the ammeter equipment 1 in the time period of 9:00-9:15 are obtained. In practical application, because the single message transmission capacity is limited, one task data may be transmitted for multiple times, and then all data to be used corresponding to the service needs to be combined.
On the basis of the embodiment, the protocols among the modules can be uniformly managed so as to perform data analysis processing based on the uniformly managed rules.
Specifically, in the program of the containerized distributed node, the method comprises a message data analysis module, a service data conversion module and a service data merging module, wherein an internal protocol is used among the three modules, so that the modules can directly judge whether the data to be collected and processed are needed, the message is prevented from being analyzed, repeated step operation is reduced, and the matching efficiency among the modules is improved. Fig. 3 is a diagram of a data processing module according to an embodiment of the present invention, where, as shown in fig. 3, a correspondence may be established in an internal protocol for each field, a field length, and a field specific name.
In the embodiment of the invention, the message data analysis module, the service data conversion module and the service data merging module correspond to three data processing processes, and the data receiving and transmitting among the three modules are realized by using an asynchronous stream processing mode, so that serial processing is changed into concurrent waterfall processing, and the data processing efficiency is further improved.
According to the technical scheme, the data to be processed acquired by mass terminal equipment are received; analyzing the data to be processed to obtain data to be stored in a target format; converting the data to be stored according to preset rules to obtain data to be used; and merging the data to be used to obtain target storage data. The containerized distributed nodes are deployed on the computer cluster, and the data to be processed uploaded by the mass terminal equipment can be analyzed, converted and combined based on the distributed nodes, so that the mass data uploaded by different terminal types are processed in real time, the data processing capacity is enlarged, and the data processing efficiency is improved.
Example two
Fig. 4 is a flowchart of a method for collecting and processing containerized distributed data according to a second embodiment of the present invention, on the basis of the foregoing embodiment, the data to be stored may be converted according to a preset rule to obtain data to be used, and the data to be used is combined to obtain target storage data for further refinement, and a specific implementation manner of the method may be referred to the detailed description of the embodiment of the present invention, where the technical terms identical to or corresponding to the foregoing embodiment are not repeated herein.
As shown in fig. 4, the method includes:
s210, receiving data to be processed acquired by mass terminal equipment.
S220, analyzing the data to be processed to obtain the data to be stored in the target format.
S230, loading the metering point file and the terminal file based on the service data conversion module to determine at least one of a terminal identifier, a terminal type, an operating electric energy meter identifier, a metering point code, a terminal address and a metering point type.
The business data conversion module is a module for converting data to be processed in a program. The metering point file refers to a file comprising information related to each ammeter. The metering point profile may include: and running the electric energy meter identification, the measuring point code, the terminal address and the metering point type. The running ammeter identification refers to a character string which can uniquely identify the corresponding ammeter in the program. The measurement point code refers to a code representing an electricity meter in data to be stored. The terminal address refers to an IP address corresponding to each terminal. The metering point type refers to the type number of the electric meter, if the metering point type is 8000, the electric meter is a power station electric meter, the metering point type is 8300, and the electric meter is a resident meter which is a resident meter. The terminal profile refers to a profile including each associated information corresponding to each terminal. The terminal profile may include: terminal identification, terminal address, terminal type, etc. The terminal identifier refers to a character string that can uniquely identify the corresponding terminal within the program. The terminal type refers to the type of terminal equipment divided according to the type of the connection ammeter, and is represented by numbers in the archive. For example, 01 indicates that the type of the terminal is a station ammeter corresponding terminal; 02 denotes a terminal type corresponding to the resident electric meter.
Specifically, the service data conversion module loads the metering point file and the terminal file, and further determines a terminal identifier, a terminal address, a terminal type, an operating electric energy meter identifier, a metering point code, a terminal address and a metering point type.
Illustratively, since the meaning of each row of data representation in the metering point archive is the same, the description will now be made with the first row of data: the first row of data is [071015048 101 00108110 5] which sequentially represents the running electric energy meter identification, the measuring point code, the terminal address and the metering point type. Similarly, the first line of data [1111198521 008555515 04] in the terminal file sequentially represents the terminal identifier, the terminal address and the terminal type corresponding to a certain terminal.
S240, determining the terminal identification and the running electric energy meter identification based on the data to be stored, the metering point file and the terminal file.
Specifically, since the manner of determining each terminal identifier is the same, the manner of determining one of the terminal identifiers will now be described: traversing the terminal file according to the terminal address corresponding to the current terminal in the data to be stored, and embedding the terminal identifier corresponding to the terminal address of the current terminal in the terminal file into the corresponding position in the data to be stored. For example, the terminal identification may be embedded after the terminal address or at the end of the data to be stored corresponding to the terminal address. And carrying out the same operation on each terminal address in the data to be stored, and obtaining the corresponding terminal identification of each terminal in the program.
Further, since the manners of determining the identifiers of the running electric energy meters are the same, the description will be made in a manner of determining one of the identifiers of the running electric energy meters: traversing the metering point file according to the terminal address and the metering point code corresponding to the current electric energy meter in the data to be stored, and embedding the running electric energy meter identifier and the metering point type corresponding to the current electric energy meter in the metering point file into the corresponding position in the data to be stored. For example, the running electric energy meter identification and the metering point type may be embedded after the terminal address and the metering point code or embedded at the end of the data to be stored corresponding to the terminal address and the metering point code. And carrying out the same operation on each electric energy meter in the data to be stored, and obtaining the corresponding operation electric energy meter identification of each electric energy meter in the program.
S250, loading a task file based on a business data conversion module to determine a task to be acquired, and converting the time of the task to be acquired according to an acquisition time range corresponding to the task to be acquired to obtain target task time; wherein the acquisition time range includes at least one acquisition instant.
The task file refers to a file for storing each acquisition task and data associated with each acquisition task. For example, the data associated with each acquisition task may be, but is not limited to, acquisition task ID, terminal type, acquisition frequency, acquisition task, and the like. The task to be collected refers to the collection process of various business data. For example, the tasks to be collected may be resident day frozen power indication, minute level power indication, month frozen demand, electricity meter power curve, etc. The task collecting time refers to the initial time corresponding to the task data to be collected. The collection time range refers to the collection frequency corresponding to each collection task in the task file.
Specifically, on the containerized distributed node, a task file is loaded, and then each acquisition task in the task file is determined as a task to be acquired. And acquiring the corresponding acquisition time range of each file to be acquired from the task files.
By way of example, a task file is loaded through a service data conversion module, data to be stored corresponding to a task ID of an ammeter power curve task in the task file is searched in the data to be stored according to the task ID, the acquisition time range corresponding to the ammeter power curve task is 15 minutes, the time of the task to be acquired is 2022-11-25,9:00, and the target task time is 2022-11-25,9:00-9:15.
And S260, obtaining corresponding data to be used based on the data to be stored, the terminal identification, the operation electric energy meter identification and the target task time.
Specifically, on the basis of the data to be stored, each terminal identifier may be added to the end of the data to be stored corresponding to each terminal address, and each running electric energy meter identifier and the type of the metering point may be added to the end of the data to be stored corresponding to each terminal address and the metering point code corresponding to each electric energy meter. And meanwhile, converting the task time to be acquired corresponding to each task according to the corresponding acquisition time range, and further obtaining data to be used.
S270, determining an operation electric energy meter identifier corresponding to the task to be acquired.
Specifically, based on S260, in the to-be-used data corresponding to each to-be-collected task, an operation electric energy meter identifier corresponding to each to-be-collected task may be determined.
And S280, according to the running electric energy meter identifier, merging the data to be used at least one acquisition time corresponding to the running electric energy meter identifier, and converting the merged data into json format for output to obtain the target storage data.
Specifically, because the processing modes of the data to be used corresponding to each running electric energy meter are the same, the data to be used of one running electric energy meter will now be described: and acquiring the data to be used at each moment corresponding to the current running electric energy meter identifier, merging the data to be used corresponding to each acquisition moment in each target task time according to the target task time corresponding to each acquisition task to obtain the data to be used corresponding to each acquisition task. And converting the data to be used corresponding to each task to be acquired, which corresponds to each electric energy meter, into json format for output, and further obtaining target storage data so that a user can check task data corresponding to each electric energy meter.
According to the technical scheme, the metering point file and the terminal file are loaded based on the service data conversion module to determine at least one of a terminal identifier, a terminal address, a terminal type, an operating electric energy meter identifier, a measuring point code, a terminal address and a metering point type; determining the terminal identification and the running electric energy meter identification based on the data to be stored, the metering point file and the terminal file; loading a task file based on a service data conversion module to determine a task to be acquired, and converting the time of the task to be acquired according to the acquisition time range corresponding to the task to be acquired to obtain target task time; wherein the acquisition time range comprises at least one acquisition time; and obtaining corresponding data to be used based on the data to be stored, the terminal identification, the running electric energy meter identification and the acquisition time range. Determining an operation electric energy meter identifier corresponding to a task to be acquired; and according to the running electric energy meter identifier, merging the data to be used at least one acquisition time corresponding to the running electric energy meter identifier, converting the merged data into json format for output to obtain target storage data, acquiring the terminal identifier, the metering point identifier and the type of the metering point according to the terminal file and the metering point file, merging the acquired task data through the running electric energy meter identifier and the target task time, converting the merged data into json format data, realizing the extraction of the task data of each electric meter, improving the data processing efficiency and facilitating the user to check the electric meter data.
Example III
Fig. 5 is a schematic structural diagram of a containerized distributed data acquisition and processing device according to a third embodiment of the present invention.
As shown in fig. 5, the apparatus includes:
the data receiving module 310 is configured to receive data to be processed acquired by the mass terminal device; the data parsing module 320 is configured to parse the data to be processed to obtain data to be stored in a target format; the data conversion module 330 is configured to perform conversion processing on the data to be stored according to a preset rule, so as to obtain data to be used; and the data merging module 340 is configured to merge the data to be used to obtain target storage data.
Based on the above technical solutions, the data receiving module 310 is specifically configured to:
receiving electric energy related data collected by mass terminal equipment; the electric energy related data comprise at least one of data of a coal electric meter, a gas electric meter, a photovoltaic electric meter and the like.
Based on the above technical solutions, the data parsing module 320 is specifically configured to:
and converting the data to be processed from hexadecimal into data to be stored with a target format being json format based on a message data analysis module.
Based on the above technical solutions, the data conversion module 330 specifically includes:
the file loading unit is used for loading the metering point file and the terminal file based on the service data conversion module so as to determine at least one of a terminal identifier, a terminal address, a terminal type, an operation electric energy meter identifier, a measuring point code, a terminal address and a metering point type;
the identification determining unit is used for determining the terminal identification and the running electric energy meter identification based on the data to be stored, the metering point file and the terminal file;
the acquisition time range determining unit is used for loading a task file based on the business data conversion module to determine a task to be acquired, and converting the time of the task to be acquired according to the acquisition time range corresponding to the task to be acquired to obtain target task time; wherein the acquisition time range comprises at least one acquisition time;
and the to-be-used data determining unit is used for obtaining corresponding to-be-used data based on the to-be-stored data, the terminal identification, the running electric energy meter identification and the target task time.
Based on the above technical solutions, the data merging module 340 is specifically configured to:
determining an operation electric energy meter identifier corresponding to a task to be acquired; and according to the running electric energy meter identifier, merging the data to be used at least one acquisition time corresponding to the running electric energy meter identifier, and converting the merged data into json format for output to obtain the target storage data.
Based on the above technical solutions, the data processing apparatus may include:
and the protocol management module is used for carrying out unified management on protocols among the modules so as to carry out data analysis processing based on the rule of unified management.
According to the technical scheme, the data to be processed acquired by mass terminal equipment are received; analyzing the data to be processed to obtain data to be stored in a target format; converting the data to be stored according to preset rules to obtain data to be used; and merging the data to be used to obtain target storage data. The containerized distributed nodes are deployed on the computer cluster, and the data to be processed uploaded by the mass terminal equipment can be analyzed, converted and combined based on the distributed nodes, so that the mass data uploaded by different terminal types are processed in real time, the data processing capacity is enlarged, and the data processing efficiency is improved.
The containerized distributed data acquisition and processing device provided by the embodiment of the invention can execute the containerized distributed data acquisition and processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Example IV
Fig. 6 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the containerized distributed data acquisition processing method.
In some embodiments, the containerized distributed data acquisition processing method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the containerized distributed data acquisition processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the containerized distributed data acquisition processing method in any other suitable manner (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. The method for collecting and processing containerized distributed data is characterized by comprising the following steps of: at least one distributed container for performing the steps of:
receiving data to be processed acquired by mass terminal equipment;
analyzing the data to be processed to obtain data to be stored in a target format;
converting the data to be stored according to preset rules to obtain data to be used;
and merging the data to be used to obtain target storage data.
2. The method of claim 1, wherein the receiving the data to be processed collected by the mass terminal device comprises:
receiving electric energy related data collected by mass terminal equipment;
the electric energy related data comprise at least one of data of a coal electric meter, a gas electric meter, a photovoltaic electric meter and the like.
3. The method of claim 1, wherein the parsing the data to be processed to obtain the data to be stored in the target format includes:
and converting the data to be processed from hexadecimal into data to be stored with a target format being json format based on a message data analysis module.
4. The method of claim 1, wherein the converting the data to be stored according to a preset rule to obtain the data to be used comprises:
loading a metering point file and a terminal file based on the service data conversion module to determine at least one of a terminal identifier, a terminal address, a terminal type, an operating electric energy meter identifier, a metering point code, a terminal address and a metering point type;
determining the terminal identification and the running electric energy meter identification based on the data to be stored, the metering point file and the terminal file;
loading a task file based on a service data conversion module to determine a task to be acquired, and converting the time of the task to be acquired according to the acquisition time range corresponding to the task to be acquired to obtain target task time; wherein the acquisition time range comprises at least one acquisition time;
and obtaining corresponding data to be used based on the data to be stored, the terminal identification, the running electric energy meter identification and the target task time.
5. The method according to claim 1, wherein the merging the data to be used to obtain target storage data includes:
determining an operation electric energy meter identifier corresponding to a task to be acquired;
and according to the running electric energy meter identifier, merging the data to be used at least one acquisition time corresponding to the running electric energy meter identifier, and converting the merged data into json format for output to obtain the target storage data.
6. The method as recited in claim 1, further comprising:
and unified management is carried out on the protocols among the modules so as to carry out data analysis processing based on the unified management rules.
7. A containerized distributed data acquisition and processing device, comprising: at least one distributed container for performing the steps of:
the data receiving module is used for receiving data to be processed acquired by mass terminal equipment;
the data analysis module is used for analyzing the data to be processed to obtain data to be stored in a target format;
the data conversion module is used for carrying out conversion processing on the data to be stored according to preset rules to obtain data to be used;
and the data merging module is used for merging the data to be used to obtain target storage data.
8. The apparatus of claim 7, wherein the data receiving module comprises:
the associated data receiving unit is used for receiving the electric energy associated data acquired by the mass terminal equipment;
the electric energy related data comprise at least one of data of a coal electric meter, a gas electric meter, a photovoltaic electric meter and the like.
9. An electronic device, the electronic device comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the containerized distributed data acquisition processing method of any one of claims 1-6.
10. A computer readable storage medium storing computer instructions for causing a processor to perform the method of containerized distributed data acquisition processing of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211699476.XA CN116193296A (en) | 2022-12-28 | 2022-12-28 | Method, device, equipment and medium for collecting and processing containerized distributed data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211699476.XA CN116193296A (en) | 2022-12-28 | 2022-12-28 | Method, device, equipment and medium for collecting and processing containerized distributed data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116193296A true CN116193296A (en) | 2023-05-30 |
Family
ID=86431875
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211699476.XA Pending CN116193296A (en) | 2022-12-28 | 2022-12-28 | Method, device, equipment and medium for collecting and processing containerized distributed data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116193296A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117170816A (en) * | 2023-09-19 | 2023-12-05 | 中科驭数（北京）科技有限公司 | DPU-based containerized data acquisition method, system and deployment method |
-
2022
- 2022-12-28 CN CN202211699476.XA patent/CN116193296A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117170816A (en) * | 2023-09-19 | 2023-12-05 | 中科驭数（北京）科技有限公司 | DPU-based containerized data acquisition method, system and deployment method |
| CN117170816B (en) * | 2023-09-19 | 2024-10-18 | 中科驭数（北京）科技有限公司 | DPU-based containerized data acquisition method, system and deployment method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116225769B (en) | Method, device, equipment and medium for determining root cause of system fault | |
| CN116193296A (en) | Method, device, equipment and medium for collecting and processing containerized distributed data | |
| CN110380902B (en) | Topological relation generation method and device, electronic equipment and storage medium | |
| CN116303761A (en) | Distributed warehousing method and device, electronic equipment and storage medium | |
| CN115687406A (en) | Sampling method, device and equipment of call chain data and storage medium | |
| CN115934550A (en) | Test method, test device, electronic equipment and storage medium | |
| CN117394867A (en) | Data compression method and device, electronic equipment and storage medium | |
| CN117742900B (en) | Method, device, equipment and storage medium for constructing service call graph | |
| CN118055051B (en) | Interface communication rate detection control method, device, equipment and medium | |
| CN115208871A (en) | Hydrogen production equipment monitoring system and method and storage medium | |
| CN117131315B (en) | Out-of-tolerance electric energy meter determining method and medium based on solving multi-element quadratic function extremum | |
| CN117971815A (en) | Method, device, equipment and medium for collecting and sorting transformer monitoring data | |
| CN117421377A (en) | Data processing method, device, equipment and medium for energy station | |
| CN117632977A (en) | Method and device for checking real-time library data, electronic equipment and storage medium | |
| CN115567624A (en) | Message processing method and device, electronic equipment and medium | |
| CN113220524A (en) | Monitoring system, method and device of micro server and electronic equipment | |
| CN118418132A (en) | State monitoring method, device, equipment and system of photovoltaic panel installation robot | |
| CN116506055A (en) | Method, device, equipment and medium for determining time tag | |
| CN118861882A (en) | Method, device, equipment and storage medium for judging time period category | |
| CN117061412A (en) | Transmission path determining method and device in power system and electronic equipment | |
| CN115455692A (en) | Method, device, equipment and medium for pushing radar digital twin data | |
| CN118914958A (en) | Method for detecting shunt abnormality of three-phase electric energy metering device, electronic equipment and medium | |
| CN116225568A (en) | Management method, device and equipment of application system interface and storage medium | |
| CN118193027A (en) | Firmware associated information updating method, device, equipment and medium | |
| CN117131990A (en) | Power grid infrastructure information management method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |