CN111694828A - Data processing method, device, system and storage medium - Google Patents
Data processing method, device, system and storage medium Download PDFInfo
- Publication number
- CN111694828A CN111694828A CN202010510922.2A CN202010510922A CN111694828A CN 111694828 A CN111694828 A CN 111694828A CN 202010510922 A CN202010510922 A CN 202010510922A CN 111694828 A CN111694828 A CN 111694828A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- data processing
- data acquisition
- format
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 61
- 238000006243 chemical reaction Methods 0.000 claims abstract description 29
- 238000000034 method Methods 0.000 claims abstract description 22
- 238000004806 packaging method and process Methods 0.000 claims abstract description 6
- 238000012795 verification Methods 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 6
- 238000005538 encapsulation Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 4
- 238000005859 coupling reaction Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 206010063385 Intellectualisation Diseases 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000474 nursing effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a data processing method, equipment, a system and a storage medium. The method is applied to a data acquisition and conversion module and comprises the following steps: acquiring data to be processed; based on a protocol cache data format, packaging the data to be processed; and sending the encapsulated data to a data acquisition center. The acquired data to be processed is packaged based on the protocol cache data format, the packaged data is submitted to the data acquisition center for processing, and the data received by the data acquisition center are all in the protocol cache data format, so that the data processing efficiency of the smart city can be effectively improved, and the processing requirement of the smart city in the modern society on mass data is met.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data processing method, device, system, and storage medium.
Background
Along with the development of global urbanization, the treatment difficulty of cities is increased sharply, and urban intellectualization is an important means for improving urban treatment capability. In order to realize urban intellectualization or intelligent cities, data processing is required to be fully utilized, namely, a large amount of data generated by people, facilities, systems and the like in the cities are collected and analyzed through an informatization technical means, and intelligent decision is made according to the data, so that efficient and convenient service is provided for urban residents, and the operation efficiency and treatment capacity of the whole city are improved.
At present, the data processing scheme in smart cities is as follows: the data acquisition equipment submits the acquired data to a data acquisition center one by one, and the data acquisition center forwards or processes the data after analyzing the data. The inventors have found that this data processing has at least the following problems:
the smart city generates massive data every day, and a data acquisition system needs to be continuously adjusted to receive data from various sources, so that the data processing efficiency of the smart city is low.
Disclosure of Invention
The invention provides a data processing method, equipment, a system and a storage medium, which can effectively improve the data processing efficiency of a smart city.
In a first aspect, the present invention provides a data processing method, including:
acquiring data to be processed;
based on a protocol cache data format, packaging data to be processed;
and sending the encapsulated data to a data acquisition center.
Optionally, acquiring data to be processed includes:
receiving and analyzing original data;
verifying the analyzed data according to a preset verification rule;
and determining the data which is successfully verified as the data to be processed.
Optionally, before encapsulating the data to be processed based on the protocol cache data format, the method further includes:
serializing the data to be processed into a byte stream by adopting a preset serialization algorithm;
correspondingly, the encapsulating the data to be processed based on the protocol cache data format comprises: and encapsulating the byte stream based on a protocol cache data format.
Optionally, after the analyzed data is verified according to the preset verification rule, the data processing method further includes:
and discarding the data failed in the verification.
Optionally, the preset serialization algorithm includes: object profile JSON and/or data serialization system Avro.
Optionally, the data acquisition and conversion module and the source of the data to be processed are located in the same network.
In a second aspect, the present invention provides a data processing apparatus comprising:
the acquisition unit is used for acquiring data to be processed;
the processing unit is used for packaging the data to be processed based on a protocol cache data format;
and the sending unit is used for sending the data after the sequence encapsulation to the data acquisition center.
Optionally, the obtaining unit is specifically configured to:
receiving and analyzing original data;
verifying the analyzed data according to a preset verification rule;
and determining the data which is successfully verified as the data to be processed.
Optionally, the processing unit is specifically configured to: serializing the data to be processed into a byte stream by adopting a preset serialization algorithm; the byte stream is encapsulated based on the protocol cache data format.
Optionally, the obtaining unit is further configured to: and discarding the data failed in the verification.
Optionally, the preset serialization algorithm includes: object profile JSON and/or data serialization system Avro.
Optionally, the data acquisition and conversion module and the source of the data to be processed are located in the same network.
In a third aspect, the present invention provides a data processing apparatus comprising:
a memory for storing program instructions;
a processor for invoking and executing program instructions in a memory to perform a method as claimed in any of the first aspects.
In a fourth aspect, the present invention provides a data processing system comprising:
a data acquisition and conversion module for implementing the method of any one of the first aspect;
the system comprises the Internet of things equipment and/or an informatization system, a data processing system and a data processing system, wherein the Internet of things equipment and/or the informatization system are used for providing original data;
and the data acquisition center is used for storing data.
In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon; the computer program, when executed by a processor, implements the method as set forth in any one of the first aspect.
The invention provides a data processing method, a device, a system and a storage medium, firstly acquiring data to be processed; and then, based on the protocol cache data format, packaging the data to be processed, and sending the packaged data to a data acquisition center. The acquired data to be processed is packaged based on the protocol cache data format, and the packaged data is submitted to the data acquisition center for processing, so that the data received by the data acquisition center are all in the protocol cache data format, the data acquisition center and specific service scenes and requirements are decoupled, and service change influence is reduced, so that the effect of improving the data processing efficiency of the smart city is achieved, and the processing requirement of the smart city in the current society on mass data is met.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a sub-flowchart of a data processing method according to another embodiment of the present invention;
FIG. 3 is a flow chart of a data processing method according to another embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data processing apparatus according to another embodiment of the present invention;
FIG. 6 is a block diagram of a data processing system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "upper", "lower", "front", "rear", and the like, indicate orientations or positional relationships based on those shown in the drawings, are merely for convenience in describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention. In the description of the invention, "a plurality" means two or more unless specifically stated otherwise.
The terms "first," "second," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises" and "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such article or apparatus.
The description includes reference to the accompanying drawings, which form a part hereof. The figures show diagrams in accordance with exemplary embodiments. These embodiments, which may also be referred to herein as "examples," are described in sufficient detail to enable those skilled in the art to practice embodiments of the claimed subject matter described herein. The embodiments may be combined, other embodiments may be utilized, or structural, logical, and electrical changes may be made without departing from the scope and spirit of the claimed subject matter. It should be appreciated that the embodiments described herein are not intended to limit the scope of the subject matter, but rather to enable any person skilled in the art to practice, make, and/or use the subject matter.
Generally, a data processing scheme of a smart city is that data acquisition equipment acquires massive, heterogeneous and scattered city data and submits the data to a data acquisition center. Because cities have a large amount of data existing in various modes and the cities can continuously generate real-time running state data, huge network bandwidth can be occupied, and because the data sources are different and the data formats are different, the data acquisition center needs to make corresponding adjustment to adapt to various data formats, so that the data processing efficiency of the data acquisition center is influenced, and the processing requirement of the mass data of the cities at present cannot be met.
Based on the problems in the prior art, embodiments of the present invention provide a data processing method, device, system and storage medium, which decouple a data processing center and specific service scenarios and requirements by unifying the format of data sent to the data acquisition center, thereby achieving the effect of improving the data processing efficiency of a smart city.
The data processing scheme provided by the present invention is explained in detail by specific examples below.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention. The embodiment of the invention provides a data processing method, which is applied to a data acquisition and conversion module, and the module can be realized in a hardware and/or software mode. The data processing method comprises the following steps:
and S10, acquiring the data to be processed.
The data to be processed may include, but is not limited to, a data receiving timestamp, data source information, and the like.
This step can be achieved in a number of ways. In one implementation, the steps may include: and receiving data sent by the Internet of things equipment and/or the information system. For example, a smart city system has branches in many locations, each branch has data related to local area in its database, and the branches will actively send data to the data collection and conversion module continuously. In another implementation manner, the data acquisition and conversion module may acquire the data to be processed by performing bypass interception on the network where the internet of things device and/or the information device is located.
It can be understood that, for different internet of things devices and/or informatization systems, the data formats of the corresponding to-be-processed data may be the same or different.
And S20, encapsulating the data to be processed based on a Protocol Buffer (PB) data format.
The purpose of this step is to unify the data format. In consideration of the fact that data formats generated by different internet of things devices and/or information systems are diversified, for example, heterogeneous, in this step, after the to-be-processed data is acquired, the data formats of the to-be-processed data are unified, that is, the to-be-processed data is encapsulated based on the protocol cache data format, so that the strong coupling relationship between the data acquisition center and the data source is decoupled.
The protocol cache data format is a data description Language, is similar to an extensible markup Language (XML for short), can serialize structured data, can be used for the aspects of data storage, communication protocols and the like, is independent of languages and platforms, and has extremely strong extensibility. The step is based on a protocol cache data format, can define the structure of data to be processed, and generates codes based on various languages. On one hand, the advantages of excellent serialization performance and small data volume of a protocol cache data format are fully utilized; on one hand, flexible adjustment of the format and the content of the service data is supported, and development and maintenance cost of the data acquisition center is reduced.
And S30, sending the packaged data to a data acquisition center.
The data acquisition and conversion module sends the packaged data to the data acquisition center, and the data acquisition center further processes the data.
In this embodiment, the data acquisition and conversion module first acquires data to be processed, encapsulates the data to be processed based on a protocol cache data format, and further sends the encapsulated data to the data acquisition center. The acquired data to be processed is packaged based on the protocol cache data format, and the packaged data is submitted to the data acquisition center for processing, so that the data received by the data acquisition center are all in the protocol cache data format, the data acquisition center and specific service scenes and requirements are decoupled, and service change influence is reduced, so that the effect of improving the data processing efficiency of the smart city is achieved, and the processing requirement of the smart city in the current society on mass data is met.
Fig. 2 is a sub-flowchart of a data processing method according to another embodiment of the present invention. Referring to fig. 2, based on the flow shown in fig. 1, the step S10 of acquiring data to be processed may include:
s101, receiving and analyzing original data.
Wherein the raw data may include heterogeneous data and/or homogeneous data. The raw data is data directly generated or obtained by the internet of things equipment and/or the informatization system. And after the data acquisition and conversion module receives the original data, analyzing the original data, and enabling the original data to be easier to read and write through analysis.
And S102, verifying the analyzed data according to a preset verification rule.
Further, the data acquisition and conversion module checks the analyzed data according to a preset check rule. For example, the preset verification rule may include: and checking according to a data mode (English: Schema) of the data, wherein the data mode comprises the composition, the name, the type, the value range and the like of the data. By checking the data mode of the data, it can be checked whether the data meets the requirements of security and/or integrity.
S103, determining the data which is successfully verified as the data to be processed.
If the verification is successful, the data is determined to be the data to be processed, and the data processing flow shown in fig. 1 is performed.
In this embodiment, the data acquisition and conversion module receives and analyzes the original data, verifies the analyzed data according to a preset verification rule, and determines that the successfully verified data is the data to be processed. The original data are analyzed, so that the original data are easier to read and write, the data acquisition and conversion module is convenient to check, the data meeting the requirements of safety and/or integrity and the like are screened as the data to be processed after the data are checked, only the data which are successfully checked are sent to the data processing center, the processing pressure and the data processing amount of the data acquisition center are reduced, and the data processing efficiency of the smart city system is improved; in addition, the data transmission pressure can be reduced, and the requirements of network bandwidth and the data processing pressure and data volume of a data acquisition center can be reduced.
In addition, when the service requirement changes, the service data can be correspondingly changed according to the construction requirement of the smart city, only the targeted maintenance development of the data acquisition and conversion module is needed, the data acquisition center does not need any adjustment, and the operation and maintenance cost of the smart city can be obviously reduced.
Fig. 3 is a flowchart of a data processing method according to another embodiment of the present invention. Referring to fig. 3, the data processing method may include:
s301, receiving and analyzing original data.
And S302, verifying the analyzed data according to a preset verification rule.
And S303, determining the successfully verified data as the data to be processed.
S301 to S303 are similar to S101 to S103 in the embodiment shown in fig. 2, and specific description may refer to the embodiment shown in fig. 2, which is not repeated herein;
s304, discarding the data which fails in verification.
It can be understood that, in this embodiment, when the data check fails, the data is discarded, and the data processing flow of the data is ended, so that the data amount of the subsequent data processing can be reduced.
S305, serializing the data to be processed into a byte stream by adopting a preset serializing algorithm.
The pre-serialization algorithm may include: object Notation format (JSON for short) and/or serialization system Avro, etc.
Corresponding to S305, as S20 shown in fig. 1, the data to be processed is encapsulated based on the protocol cache data format, where the following is replaced by: s306, based on the protocol cache data format, the byte stream is packaged.
The data acquisition and conversion module assembles information such as byte streams, current time stamps, data providers and the like into a protocol cache data format in batch.
And S307, sending the encapsulated data to a data acquisition center.
In practical application, the data acquisition and conversion module can transmit the encapsulated data to the data acquisition center through a specific interface. The specific interface may include, but is not limited to, a REST (Representational State Transfer, abbreviated as REST) interface, and the like. If the encapsulated data is sent to the data acquisition center through the REST interface, the data acquisition center only needs to define a uniform response interface.
In any of the above embodiments, the data collection and conversion module and the source of the data to be processed are located in the same network. It can be understood that each internet of things device or each informatization system respectively corresponds to at least one data acquisition and conversion module, and a one-to-one or one-to-many relationship is formed. Therefore, when a new data format appears, only the data acquisition and conversion module corresponding to the Internet of things equipment (or the information system) generating the new data format needs to be updated, and therefore maintenance cost is reduced.
Fig. 4 is a schematic structural diagram of a data processing device according to an embodiment of the present invention. Embodiments of the present invention provide a data processing device, which may be implemented in a software and/or hardware manner. Referring to fig. 4, the data processing apparatus 40 includes: an acquisition unit 41, a processing unit 42 and a transmission unit 43. Wherein the content of the first and second substances,
an obtaining unit 41, configured to obtain data to be processed.
And the processing unit 42 is used for encapsulating the data to be processed based on the protocol cache data format.
And the sending unit 43 is used for sending the encapsulated data to the data acquisition center.
The data processing apparatus provided in this embodiment may be configured to execute the method embodiments described above, and the implementation manner and the technical effect of the data processing apparatus are similar, which is not described herein again.
Optionally, the obtaining unit 41 may be specifically configured to: receiving and analyzing original data; verifying the analyzed data according to a preset verification rule; and determining the data which is successfully verified as the data to be processed.
Further, the processing unit 42 may be specifically configured to: serializing the data to be processed into a byte stream by adopting a preset serialization algorithm; the byte stream is encapsulated based on the protocol cache data format.
Optionally, the obtaining unit 41 is further configured to: and discarding the data failed in the verification.
Optionally, the pre-serialization algorithm may include: JSON and/or Avro, and the like.
Furthermore, the data acquisition and conversion module and the source of the data to be processed are located in the same network.
Optionally, the sending unit 43 may be specifically configured to: and sending the encapsulated data to a data acquisition center through a preset interface. In practical applications, the preset interface may be a REST interface or the like. The sending module 43 sends the packaged data through the preset interface, and the data acquisition center only needs to define a uniform response interface and does not need to analyze various heterogeneous data, thereby achieving the technical effect of improving the data processing efficiency.
Fig. 5 is a schematic structural diagram of a data processing device according to another embodiment of the present invention. Embodiments of the present invention provide a data processing device, which may be implemented in a software and/or hardware manner. Referring to fig. 5, the data processing apparatus 50 includes: memory 51, processor 52.
Wherein the memory 51 stores program instructions.
A processor 52 for calling and executing the program instructions in the memory 51, so that the processor 52 executes the data processing method as described above.
Optionally, the data processing device 50 may also include a communication bus 53. The communication bus 53 is used to connect the processor 52 and the memory 51.
FIG. 6 is a block diagram of a data processing system according to an embodiment of the present invention. Referring to fig. 6, the data processing system 60 includes: the system comprises an internet of things device 61, an informatization system 62, a data acquisition and conversion device 63 and a data acquisition center 64.
The internet of things device 61 and/or the informatization system 62 are used to provide raw data.
In practical applications, the internet-of-things device 61 and/or the informatization system 62 may include: the intelligent transportation, environmental protection, government work, public safety, safe house, intelligent fire control, industrial monitoring, environmental monitoring, old man nursing, personal health, flower cultivation, water system monitoring, food tracing, enemy investigation and information collection. Optionally, the internet of things device 61 and/or the informatization system 62 is implemented by embedding and installing sensors into various objects such as power grids, railways, bridges, tunnels, highways, buildings, water supply systems, oil and gas pipelines, so as to integrate human society with physical systems. The internet of things device 61 and/or the informatization system 62 integrates and transmits the raw data to the data acquisition and conversion module 63.
The data acquisition and conversion module 63 is connected to the internet of things device 61 and/or the information system 62, and configured to acquire original data in the internet of things device 61 and/or the information system 62, and perform data processing such as parsing, checking, and data format conversion.
And the data acquisition center 64 is used for receiving and storing the data sent by the data acquisition and conversion module 63.
The embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used for implementing the data processing method provided in any of the above embodiments when being executed by a processor.
In the above embodiments, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the modules is only one logical division, and other divisions may be realized in practice, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.
The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention.
It should be understood that the Processor may be a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the present invention may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.
The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile storage NVM, such as at least one disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic or optical disk, etc.
The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (enhanced Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present invention are not limited to only one bus or one type of bus.
The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks, and so forth. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
Those of ordinary skill in the art will understand that: all or a portion of the steps of implementing the above-described method embodiments may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A data processing method is applied to a data acquisition and conversion module, and comprises the following steps:
acquiring data to be processed;
based on a protocol cache data format, packaging the data to be processed;
and sending the encapsulated data to a data acquisition center.
2. The method of claim 1, wherein the obtaining the data to be processed comprises:
receiving and analyzing original data;
verifying the analyzed data according to a preset verification rule;
and determining the successfully verified data as the data to be processed.
3. The method of claim 1, wherein before encapsulating the to-be-processed data based on a protocol cache data format, further comprising:
serializing the data to be processed into a byte stream by adopting a preset serialization algorithm;
correspondingly, the encapsulating the data to be processed based on the protocol cache data format comprises: and encapsulating the byte stream based on a protocol cache data format.
4. The method according to claim 2, wherein after the verifying the parsed data according to the preset verification rule, the method further comprises:
and discarding the data failed in the verification.
5. The method of claim 3, wherein the pre-set serialization algorithm comprises: object profile JSON and/or data serialization system Avro.
6. The method of any one of claims 1 to 5, wherein the data acquisition and conversion module is located on the same network as the source of the data to be processed.
7. A data processing device, applied to a data acquisition and conversion module, comprises:
the acquisition unit is used for acquiring data to be processed;
the processing unit is used for packaging the data to be processed based on a protocol cache data format;
and the sending unit is used for sending the data after the sequence encapsulation to the data acquisition center.
8. A data processing apparatus, characterized by comprising:
a memory for storing program instructions;
a processor for calling and executing program instructions in said memory, performing the method of any of claims 1 to 6.
9. A data processing system, comprising:
a data acquisition and conversion module for implementing the method of any one of claims 1 to 6;
the system comprises the Internet of things equipment and/or an informatization system, a data processing system and a data processing system, wherein the Internet of things equipment and/or the informatization system are used for providing original data;
and the data acquisition center is used for storing data.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program; the computer program, when executed by a processor, implements the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010510922.2A CN111694828A (en) | 2020-06-08 | 2020-06-08 | Data processing method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010510922.2A CN111694828A (en) | 2020-06-08 | 2020-06-08 | Data processing method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111694828A true CN111694828A (en) | 2020-09-22 |
Family
ID=72479700
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010510922.2A Pending CN111694828A (en) | 2020-06-08 | 2020-06-08 | Data processing method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111694828A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112755519A (en) * | 2021-01-11 | 2021-05-07 | 北京星汉云图文化科技有限公司 | Sensor data processing method, device and system based on data platform and storage medium |
| CN112765139A (en) * | 2021-01-11 | 2021-05-07 | 浙江星汉云图人工智能科技有限公司 | Method, system and computer readable storage medium for data encapsulation |
| CN112783879A (en) * | 2021-01-11 | 2021-05-11 | 浙江星汉云图人工智能科技有限公司 | Data processing method, device and system based on data platform and storage medium |
| CN113014595A (en) * | 2021-03-15 | 2021-06-22 | 青岛弯弓信息技术有限公司 | Data encapsulation integration method and system |
| CN113382069A (en) * | 2021-06-09 | 2021-09-10 | 浙江中控技术股份有限公司 | Data transmission method, device and equipment for oil and gas pipeline and storage medium |
| CN113507530A (en) * | 2021-09-09 | 2021-10-15 | 深圳市安软慧视科技有限公司 | Data forwarding method, related system and device and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605766A (en) * | 2013-11-26 | 2014-02-26 | 国家电网公司 | Method and device for accessing power transformation data to real-time or historical database |
| US20170032003A1 (en) * | 2013-10-01 | 2017-02-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
| US20170078349A1 (en) * | 2012-04-13 | 2017-03-16 | Goldman, Sachs & Co. | Systems and methods for scalable structured data distribution |
| CN109218277A (en) * | 2017-07-07 | 2019-01-15 | 华中科技大学 | A kind of multi-protocol data acquisition method and device |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN110147356A (en) * | 2019-05-14 | 2019-08-20 | 厦门欢乐逛科技股份有限公司 | Data transmission method and device |
| US20190303487A1 (en) * | 2018-03-27 | 2019-10-03 | Paypal, Inc. | System and platform for computing and analyzing big data |
-
2020
- 2020-06-08 CN CN202010510922.2A patent/CN111694828A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170078349A1 (en) * | 2012-04-13 | 2017-03-16 | Goldman, Sachs & Co. | Systems and methods for scalable structured data distribution |
| CN108337320A (en) * | 2012-04-13 | 2018-07-27 | 高盛有限责任公司 | The system and method being distributed for expansible structural data |
| US20170032003A1 (en) * | 2013-10-01 | 2017-02-02 | Cloudera, Inc. | Background format optimization for enhanced sql-like queries in hadoop |
| CN103605766A (en) * | 2013-11-26 | 2014-02-26 | 国家电网公司 | Method and device for accessing power transformation data to real-time or historical database |
| CN109218277A (en) * | 2017-07-07 | 2019-01-15 | 华中科技大学 | A kind of multi-protocol data acquisition method and device |
| US20190303487A1 (en) * | 2018-03-27 | 2019-10-03 | Paypal, Inc. | System and platform for computing and analyzing big data |
| CN109257337A (en) * | 2018-08-28 | 2019-01-22 | 科大国创软件股份有限公司 | Internet of things equipment method for converting protocol and lamp stand operation platform based on this method |
| CN110147356A (en) * | 2019-05-14 | 2019-08-20 | 厦门欢乐逛科技股份有限公司 | Data transmission method and device |
Non-Patent Citations (2)
| Title |
|---|
| 佩里·肖: "基于ARM的嵌入式系统和物联网开发", 31 January 2020, 机械工业出版社, pages: 33 - 34 * |
| 王煜骢;陈兴蜀;罗永刚;王岳;: "NTCI-Flow:一种可扩展的高速网络流量处理框架", no. 1 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112755519A (en) * | 2021-01-11 | 2021-05-07 | 北京星汉云图文化科技有限公司 | Sensor data processing method, device and system based on data platform and storage medium |
| CN112765139A (en) * | 2021-01-11 | 2021-05-07 | 浙江星汉云图人工智能科技有限公司 | Method, system and computer readable storage medium for data encapsulation |
| CN112783879A (en) * | 2021-01-11 | 2021-05-11 | 浙江星汉云图人工智能科技有限公司 | Data processing method, device and system based on data platform and storage medium |
| CN113014595A (en) * | 2021-03-15 | 2021-06-22 | 青岛弯弓信息技术有限公司 | Data encapsulation integration method and system |
| CN113382069A (en) * | 2021-06-09 | 2021-09-10 | 浙江中控技术股份有限公司 | Data transmission method, device and equipment for oil and gas pipeline and storage medium |
| CN113507530A (en) * | 2021-09-09 | 2021-10-15 | 深圳市安软慧视科技有限公司 | Data forwarding method, related system and device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111694828A (en) | Data processing method, device, system and storage medium | |
| WO2021189954A1 (en) | Log data processing method and apparatus, computer device, and storage medium | |
| CN109800259B (en) | Data acquisition method and device and terminal equipment | |
| CN103077108B (en) | A kind of method and system browser collapse being carried out to early warning | |
| CN109787974B (en) | Message data stream generation method and device, computer equipment and storage medium | |
| CN112788270B (en) | Video backtracking method, device, computer equipment and storage medium | |
| CN111683066A (en) | Heterogeneous system integration method and device, computer equipment and storage medium | |
| CN113225339B (en) | Network security monitoring method and device, computer equipment and storage medium | |
| CN110806931A (en) | Method and device for realizing call link tracking of distributed service and electronic equipment | |
| CN111680104B (en) | Data synchronization method, device, computer equipment and readable storage medium | |
| CN114788199A (en) | Data verification method and device | |
| CN113064920A (en) | Real-time computing method and device based on Flink, computer equipment and storage medium | |
| CN112507265A (en) | Method and device for anomaly detection based on tree structure and related products | |
| US20190196842A1 (en) | Medium, change detection method, and change detection apparatus | |
| CN111858093B (en) | Message processing method, device, equipment and storage medium | |
| CN115017218A (en) | Processing method and device of distributed call chain, storage medium and electronic equipment | |
| CN109981738B (en) | Cloud server suitable for application of narrow-band Internet of things | |
| CN111400251A (en) | File access method, device, server and storage medium based on gateway | |
| CN113297598B (en) | Method, device, equipment and storage medium for embedding points at service end of express cabinet | |
| CN116192535B (en) | Cross-grade and cross-network-domain data transmission method and device | |
| CN103546463A (en) | Additional-parameter transmission method and additional-parameter transmission device of integrated information platform and application system | |
| CN104426997A (en) | Processing method and processing device for signaling trace | |
| CN113641576B (en) | Database testing method and device and terminal equipment | |
| CN117061588B (en) | Device access method, electronic device, and computer-readable storage medium | |
| CN113760564A (en) | Data processing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |