CN113656445A - Data processing method and device, electronic equipment and storage medium - Google Patents
Data processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113656445A CN113656445A CN202110991637.1A CN202110991637A CN113656445A CN 113656445 A CN113656445 A CN 113656445A CN 202110991637 A CN202110991637 A CN 202110991637A CN 113656445 A CN113656445 A CN 113656445A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- format
- data processing
- stored
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 claims abstract description 166
- 238000000034 method Methods 0.000 claims abstract description 35
- 238000004458 analytical method Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 12
- 230000006399 behavior Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 description 9
- 238000007405 data analysis Methods 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000001914 filtration Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000002776 aggregation Effects 0.000 description 2
- 238000004220 aggregation Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000008676 import Effects 0.000 description 2
- 230000014759 maintenance of location Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000000547 structure data Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method, a data processing device, electronic equipment and a storage medium, and relates to the technical field of software development. The method comprises the following steps: receiving data to be processed, wherein the data to be processed is acquired from a business server by a data acquisition module; if the data to be processed comprises first format data, converting the first format data into second format data; performing data processing on the data to be processed to obtain data to be stored; and storing the data to be stored in a real-time database of the data processing module. Therefore, the problem that data cannot be uniformly scheduled and uniformly processed due to large data volume and inconsistent data formats in the service server in the related art can be solved.
Description
Technical Field
The present invention relates to the field of big data processing technologies, and in particular, to a data processing method and apparatus, an electronic device, and a storage medium.
Background
In a practical enterprise scenario, if a data fusion system is implemented, several challenges are inevitably faced:
firstly, the dynamic property: because the data source changes constantly, the data processing system needs some corresponding policy to provide dynamically changeable table structures, table additions and subtractions.
Secondly, scalability: any distributed system must provide scalability. Because the system does not synchronize only one table, there are often a large number of data synchronization tasks in progress. How to perform unified scheduling in one cluster or a plurality of clusters and ensure the efficiency of parallel execution of tasks is a basic problem to be solved.
Third, fault tolerance: in any environment, the server cannot be assumed to be always in normal operation, and the network, the disk and the memory can be in failure. And is also a key issue that needs to be considered by the data processing system.
Fourth, isomerism: in a data fusion project, since the source and destination are different, for example, the source is MySQL and the destination is Oracle, it is possible that they are different for a field type defined standard. When synchronizing, if these differences are ignored, a series of problems are caused.
Fifth, consistency: consistency is the most fundamental problem in data fusion, and data consistency is guaranteed even if the speed of data synchronization is not considered. The bottom line of data consistency is: data is not lost first, and if a part of the data is lost, the service cannot be used generally.
In view of the above-mentioned problem in the related art that data cannot be uniformly scheduled and processed due to large data volume and inconsistent data format in the service server, no effective solution has been proposed at present.
Disclosure of Invention
Embodiments of the present invention provide a data processing method and apparatus, an electronic device, and a storage medium, so as to solve the problem in the related art that data cannot be uniformly scheduled and processed due to a large data volume and inconsistent data formats in a service server.
In order to solve the technical problem, the invention is realized as follows:
in a first aspect, an embodiment of the present invention provides a data processing method, where the method includes: receiving data to be processed, wherein the data to be processed is acquired from a business server by a data acquisition module; if the data to be processed comprises first format data, converting the first format data into second format data; performing data processing on the data to be processed to obtain data to be stored; and storing the data to be stored in a real-time database of a data processing module.
Further, after storing the data to be stored in the real-time database of the data processing module, the method further includes: receiving a data query request in the real-time database, wherein the data query request is sent to the data processing module by a data query display module; and sending the query result corresponding to the data query request to the data query display module.
Further, after receiving the data to be processed, the method further comprises: and storing the data to be processed into an offline database of the data processing module.
Further, if the data to be processed includes data in a first format, converting the data in the first format into data in a second format, including: determining a data format in the data to be processed through a grammar engine of the data processing module; and if the data to be processed comprises first format data, converting the first format data into second format data through the grammar engine.
Further, performing data processing on the data to be processed to obtain data to be stored, including: judging whether the data to be processed accords with a preset rule or not through a rule engine of the data processing module; under the condition that the data to be processed does not accord with the preset rule, performing data processing on the data to be processed to obtain data to be stored; and under the condition that the data to be processed accord with the preset rule, determining the data to be processed as the data to be stored.
Further, storing the data to be stored in the real-time database includes: and storing the data to be stored into a specified data target in the real-time database according to preset configuration information.
Further, the method further comprises: converting the user behavior event in the service server into third format data through the data acquisition module; storing the third format data into a cache database of the service server; and sending the third format data in the cache database to the data processing module every other preset period.
In a second aspect, an embodiment of the present invention additionally provides a data processing apparatus, where the apparatus includes: the system comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving data to be processed, and the data to be processed is acquired from a business server by a data acquisition module; the analysis unit is used for converting the first format data into second format data if the data to be processed comprises the first format data; the processing unit is used for carrying out data processing on the data to be processed to obtain data to be stored; and the storage unit is used for storing the data to be stored to a real-time database of the data processing module.
In a third aspect, an embodiment of the present invention additionally provides a data processing system, where the system includes a data acquisition module, a data processing module, and a data query and display module, where the data acquisition module is configured to send data to be processed, which is acquired from a service server, to the data processing module; the data processing module is used for analyzing and processing the data to be processed and then storing the data to be processed into a real-time database of the data processing module; and the data query display module is used for sending the data query request to the data processing module and displaying the data in the real-time database.
In a fourth aspect, an embodiment of the present invention additionally provides an electronic device, including: memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the data processing method according to the first aspect.
In a fifth aspect, the present invention provides a storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the steps of the data processing method according to the first aspect.
In the embodiment of the invention, data to be processed is received, wherein the data to be processed is acquired from a business server by a data acquisition module; if the data to be processed comprises first format data, converting the first format data into second format data; performing data processing on the data to be processed to obtain data to be stored; and storing the data to be stored in a real-time database of the data processing module. The data to be processed is analyzed and processed, and the data is analyzed and processed and then stored in the real-time database of the data processing module, so that the uniform analysis processing of the data in the service server is realized, and the purpose of uniform scheduling of the service data is achieved. The problem that data cannot be uniformly scheduled and processed due to the fact that data volume in a service server is large and data formats are inconsistent in the related technology is solved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without inventive labor.
Fig. 1 is a schematic view of an application scenario of a data processing method in an embodiment of the present invention;
FIG. 2 is a flow chart of a data processing method in an embodiment of the invention;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
FIG. 4 is a block diagram of a data processing system in an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Before introducing the technical solution of the present invention, an application environment of the embodiment of the present invention is described below:
the embodiment of the present invention provides a data processing method, which applies a real-time data processing framework shown in fig. 1, and the framework includes a data acquisition module 10, a data processing module 20, and a data query display module 30, wherein the data acquisition module 10 is located in a service server 100, the data processing module 20 is located in a data processing server 200, and the data query display module 30 is located in a user terminal 300.
Specifically, the data acquisition module 10 is located at the frontmost end of the whole data processing system, and resides in the service server 100, and performs preliminary processing on the service data to be collected. The data to be processed initially collected by the data acquisition module 10 is sent to the data processing module 20, and the operations such as designated analysis, processing and data encapsulation are performed, so that scattered original data are aggregated into related indexes, or are filtered by the screening of designated rules. And finally, storing the processed data to be processed into the real-time database of the data processing module 20. The data query module 30 sends the data query request to the data processing module 20 according to the operation instruction of the user terminal, and the data processing module 20 receives the data query request for the real-time database, returns a corresponding data query result to the data query module 30, and displays the data query result in the user terminal 300.
Through the embodiment, the data to be processed is analyzed and processed, and after the data is analyzed and processed, the data is stored in the real-time database of the data processing module, so that the uniform analysis processing of the data in the service server is realized, and the purpose of uniform scheduling of the service data is achieved. The problem that data cannot be uniformly scheduled and processed due to the fact that data volume in a service server is large and data formats are inconsistent in the related technology is solved.
In an embodiment of the present invention, a data processing method is provided, as shown in fig. 2, the method may specifically include the following steps:
s202, receiving data to be processed, wherein the data to be processed is acquired from a business server by a data acquisition module;
in this embodiment, the data acquisition module is located at the frontmost end of the whole data processing system, that is, resides in the service server, and the data acquisition module performs preliminary processing on the service data and the related information to be acquired by monitoring the service request received by the service server, and then sends the acquired data to the data acquisition module.
Specifically, the data to be processed includes, but is not limited to, user behavior events generated in the service server, system service logs, configuration information, and the like. The related information of the service data includes, but is not limited to, an event type, a location, an event, a user ID, and the like.
S204, if the data to be processed comprises first format data, converting the first format data into second format data;
specifically, in this embodiment, the data format of the data to be processed is determined, and then the processing mode corresponding to the data to be processed is determined according to the data format of the data to be processed.
In some examples, the data format of the data to be processed, which is not supported by the data processing module, may be converted into a data format compatible with the data processing module. In other examples, some key values in the data format of the data to be processed are missing, and the data processing module supplements the missing key values according to the preset data format.
Specifically, in this embodiment, after the data format of the data to be processed is determined, the data to be processed is subjected to data parsing, and specifically, the data parsing manner includes, but is not limited to, parsing the data format, the data source, the data target, the configuration information, and the like of the data to be processed.
For example, the data processing module uniformly converts the unrecognized data format into the data format recognizable by the data processing module; the data processing module determines a data source of the data to be processed, and then determines a storage position of the data to be processed, a data processing mode and the like.
In this embodiment, after determining the data format of the data to be processed, if the data to be processed includes the first format data, the data conversion is performed on the first format data. The first data format is a data format which cannot be identified or preset by the data processing module, when the data to be processed comprises the data in the first data format, the data in the first format is converted into second-type data, and the data in the second format is a data format which can be identified and processed by the data processing module.
Specifically, in this embodiment, the first format data is a heterogeneous data format, a heterogeneous data source, and a heterogeneous data target, and the data processing module converts the first format data into the second format data, that is, a data format with the same structure as the data processing module. In this embodiment, the data processing module can read the data meaning of the data in the second format, and then perform processing modes such as filtering, replacing, repairing, dereferencing, calculating, and key changing on the data to be processed according to a preset rule to obtain the data to be stored, and then store the processed data to be stored in the real-time database.
S206, performing data processing on the data to be processed to obtain data to be stored;
on the other hand, the data processing mode of the data to be processed includes, but is not limited to, filtering, replacing, etc. the data to be processed. By interpreting a piece of configuration information, actual processing of a piece of data is realized, such as filtering by a specified value (for example, data with retention date greater than 20210301, other discarding), and for example, replacing a key value of original data, such as key1 of original data to key 2.
And S208, storing the data to be stored into a real-time database of the data processing module.
And then, after data analysis and data processing are carried out on the data to be processed, the processed data to be processed is stored in a real-time database of the data processing module, and the real-time database is located in the data processing server. The real-time database is a high-performance real-time database, and in this embodiment, the data in the service server is processed and aggregated in real time by the data processing module and then stored in the high-performance real-time database.
It should be noted that, in the embodiment of the present invention, to-be-processed data is received, where the to-be-processed data is acquired from a service server by a data acquisition module; if the data to be processed comprises first format data, converting the first format data into second format data; performing data processing on the data to be processed to obtain data to be stored; and storing the data to be stored in a real-time database of the data processing module. The data to be processed is analyzed and processed, and the data is analyzed and processed and then stored in the real-time database of the data processing module, so that the uniform analysis processing of the data in the service server is realized, and the purpose of uniform scheduling of the service data is achieved. The problem that data cannot be uniformly scheduled and processed due to the fact that data volume in a service server is large and data formats are inconsistent in the related technology is solved.
Optionally, in this embodiment, after storing the data to be stored in the real-time database of the data processing module, the method further includes, but is not limited to: receiving a data query request in a real-time database, wherein the data query request is sent to a data processing module by a data query display module; and sending the query result corresponding to the data query request to the data query display module.
Specifically, in the embodiment, real-time data use capabilities such as real-time query, data bulletin board, real-time SQL analysis, and the like can be realized in the data query display module by using the aggregated basic data stored in the high-performance real-time database.
Optionally, in this embodiment, after receiving the data to be processed, the method further includes, but is not limited to: and storing the data to be processed into an offline database of the data processing module.
Specifically, in this embodiment, the original information primarily acquired by the data acquisition module is copied into two copies, and the two paths are processed respectively, where one copy of data is used as the original data detail and is directly sent to an offline database, for example, an HDFS distributed storage system. The partial data is used as an original backup for later data analysis. And the other data is sent to the data processing module, and designated analysis, processing, packaging and other operations are performed, and scattered original data is aggregated into related indexes, such as the number of times that a certain user browses certain types of commodities in the week, or screening and filtering through designated rules, and the like. And finally, storing the data to be processed by the data processing module into a real-time database.
According to the embodiment, the original data to be processed in the data to be processed collected by the data collection module is stored in the off-line database, the processed data to be processed is stored in the real-time database, and the data capacity support can be provided for subsequent data report analysis and real-time data large-scale disc by means of the support of the two databases.
Optionally, if the data to be processed includes data in a first format, the data in the first format is converted into data in a second format, which includes but is not limited to: determining a data format in the data to be processed through a grammar engine of the data processing module; and if the data to be processed comprises the data in the first format, converting the data in the first format into the data in the second format through the grammar engine.
Specifically, a grammar engine is provided in the data processing module, and the grammar engine is used for parsing the data to be processed into an actual data meaning, for example, the grammar engine parses a piece of configuration information into the actual meaning, and applies the configuration information to a subsequent data processing process of the data processing module. In practical application, the grammar engine includes functions of numerical value reading, expression assembling and the like, and is used for performing numerical value reading and expression assembling on data to be processed, converting first format data in the data to be processed into second format data, namely converting a heterogeneous data format into data in the same format as that in a data processing mode, and further acquiring data meaning of the data to be processed.
It should be noted that the first format data includes, but is not limited to, heterogeneous data formats, heterogeneous data sources, and heterogeneous data targets waiting for processing data. In this embodiment, dynamic control of data to be processed from import to processing to output is controlled by modifying the configuration. For a certain data processing task, the configuration table records relevant information (such as account number and password of a database, theme of kafka, id of a message bus and the like) of a data source and a data target, and data analysis processing rules. And then converting the first format data into second format data, wherein the second format data comprises but is not limited to the same structure data format, the same structure data format and the data target waiting processing data with the same structure, and the second format data comprises the corresponding actual meaning of the first format data.
In one example, the grammar engine is responsible for parsing a piece of configuration information into actual meaning and passing it to the rules engine for execution. Typically, the grammar engine should include capabilities for value reading, expression packing, and the like. For example, the definition "j # key1.key 2" is expressed as reading the value of key1 under multiple layers of Json and turning to Json again, then reading the value of key2 under new Json. As another example, the definition "jfix # key 1" indicates that the value of the key1 key is repaired by the json structure, and so on.
Optionally, in this embodiment, the data to be processed is subjected to data processing to obtain data to be stored, which includes but is not limited to: judging whether the data to be processed accords with a preset rule or not through a rule engine of the data processing module; under the condition that the data to be processed does not accord with the preset rule, performing data processing on the data to be processed to obtain data to be stored; and under the condition that the data to be processed accord with the preset rule, determining the data to be processed as the data to be stored.
Specifically, in order to support heterogeneous data formats, heterogeneous data sources, and heterogeneous data targets, a configuration submodule is designed in the data processing module, and the configuration submodule controls dynamic control of data to be processed from import to processing to output in a configuration modification manner.
In one example, for a certain data processing task, the configuration table records related information of a data source and a data target (such as an account number password of a database, a subject of kafka, an ID of a message bus, and the like) and preset rules, wherein the preset rules comprise a data parsing rule and a data processing rule.
In a particular application scenario, the data to be processed is converted by the grammar engine into a data format recognizable by the rules engine. And further, performing data processing on the data to be processed by the rule engine according to the configured preset rule, and performing data processing on the data to be processed under the condition that the data to be processed does not accord with the preset rule to obtain the data to be stored. On the other hand, under the condition that the data to be processed accord with the preset rule, the data to be processed is not processed, and the data to be processed is directly used as the data to be stored.
In this embodiment, the rule engine performs data analysis and data processing on the data to be processed by reading the configuration information in the configuration table and according to the data analysis rule and the data processing rule in the configuration information. For example, filtering according to a specified value, retaining data with the date greater than 20210301, and discarding other data with the date less than or equal to 20210301; for another example, the key value of the original data is replaced, and the key1 of the original data is replaced by the key 2.
Through the grammar engine and the rule engine in the data processing module in the embodiment, the data analysis and the data processing of the data to be processed are realized through stream processing, and the low-delay processing retention of the data to be processed is realized.
Optionally, in this embodiment, the data to be stored is stored in a real-time database, which includes but is not limited to: and storing the data to be stored into a specified data target in the real-time database according to the preset configuration information.
Specifically, in this embodiment, further, in this embodiment, reading, comparing, adding and deleting fields of the designated field of the original data information is implemented by reading corresponding configuration information through a syntax engine and a rule engine in the data processing module. And further, the information is filtered and packaged again, and the index characteristics of real-time data in window time can be calculated by utilizing the aggregation capability of a Flink framework.
In addition, in this embodiment, data that is processed in one data stream may be sent to a designated data target area, such as a database, a message middleware, or a service interface, according to the configuration information.
Optionally, in this embodiment, the method further includes, but is not limited to: converting the user behavior event in the service server into third format data through a data acquisition module; storing the third format data into a cache database of the service server; and sending the third format data in the cache database to the data processing module every other preset period.
Specifically, the data acquisition module is configured to format a user behavior event occurring in the service server in real time into a specified format, and send the specified format from the service server to the data processing module.
Firstly, a data acquisition module converts a user behavior event in a service server into data in a third format, and relevant information of each user behavior event is obtained by monitoring a service request received by the service server. The related information of the user behavior event includes, but is not limited to, an event type, a location, an event, a user ID, and the like, and the related information is converted into a specified third format.
Secondly, because the data acquisition module needs to reside in the service server, the acquired information needs to be temporarily stored in a cache database in the service server after data formatting for being sent to the data processing module.
In one example, the mode of temporarily storing the data has multiple choices, and the information can be printed to a designated log file by using a log frame of the service server; in another example, the information may also be printed line by line to a specified file under a specified path.
And finally, arranging a data sending agent in the data acquisition module, observing the content increase condition of the data storage file at regular time through the data sending agent, sending the incremental data into a message middleware in an incremental sending mode at intervals of a preset period, and sending the incremental data to the data processing module through the message middleware.
It should be noted that, the third format data is obtained by performing preliminary processing on the service data requiring the mobile phone in the service server by the data acquisition module, and the processing includes formatting, recording, sending and other steps. Various scattered event information is marked in a unified format, and then unified third format data is temporarily stored in a service server file to form a temporary record. The information sending function is responsible for sending the event information temporarily stored in the service server file to the data processing system, so that the related information enters the subsequent data processing flow.
In an actual application scenario in this embodiment, the real-time database and the offline database related in this embodiment both have real-time delay-free data increments facing the data stream processed in real time in this embodiment. Based on the real-time retained data, the following functions are executed through the data query display module:
1) external real-time service: by utilizing the basic data after real-time processing and aggregation stored in the high-performance database, the data query and display module can support real-time data use capabilities such as real-time query, data bulletin board, real-time sql analysis and the like.
2) Offline analysis capability: because the raw data details which are not processed are stored in the offline database, the method can be used for offline analysis, data mining or algorithm model training of analysts.
Through the embodiment, the data to be processed is received, wherein the data to be processed is acquired from the business server by the data acquisition module; if the data to be processed comprises first format data, converting the first format data into second format data; performing data processing on the data to be processed to obtain data to be stored; and storing the data to be stored in a real-time database of the data processing module. The data to be processed is analyzed and processed, and the data is analyzed and processed and then stored in the real-time database of the data processing module, so that the uniform analysis processing of the data in the service server is realized, and the purpose of uniform scheduling of the service data is achieved. The problem that data cannot be uniformly scheduled and processed due to the fact that data volume in a service server is large and data formats are inconsistent in the related technology is solved.
Example two
A data processing apparatus according to an embodiment of the present invention is described in detail.
Referring to fig. 3, a schematic structural diagram of a data processing apparatus in an embodiment of the present invention is shown.
The data processing device of the embodiment of the invention comprises: receiving section 30, analyzing section 32, processing section 34, and storage section 36.
The functions of the modules and the interaction relationship between the modules are described in detail below.
The receiving unit 30 is configured to receive data to be processed, where the data to be processed is acquired from a service server by a data acquisition module;
the analysis unit 32 is configured to convert the first format data into second format data if the to-be-processed data includes first format data;
the processing unit 34 is configured to perform data processing on the data to be processed to obtain data to be stored;
and the storage unit 36 is used for storing the data to be stored in the real-time database of the data processing module.
Moreover, through the above embodiment, the data to be processed is received, wherein the data to be processed is acquired from the service server by the data acquisition module; if the data to be processed comprises first format data, converting the first format data into second format data; performing data processing on the data to be processed to obtain data to be stored; and storing the data to be stored in a real-time database of the data processing module. The data to be processed is analyzed and processed, and the data is analyzed and processed and then stored in the real-time database of the data processing module, so that the uniform analysis processing of the data in the service server is realized, and the purpose of uniform scheduling of the service data is achieved. The problem that data cannot be uniformly scheduled and processed due to the fact that data volume in a service server is large and data formats are inconsistent in the related technology is solved.
EXAMPLE III
Preferably, the embodiment of the present invention further provides a data processing system, as shown in fig. 4, the data processing system includes a data acquisition module 40, a data processing module 42, and a data query presentation module 44, wherein,
the data acquisition module 40 is configured to send to-be-processed data acquired from the service server to the data processing module;
the data processing module 42 is configured to perform data analysis and data processing on the data to be processed, and store the data to be processed in a real-time database of the data processing module;
and the data query display module 44 is configured to send a data query request to the data processing module, and display data in the real-time database.
When executed, the data processing system realizes each process of the data processing method embodiment, can achieve the same technical effect, and is not repeated here to avoid repetition.
Example four
Preferably, an embodiment of the present invention further provides an electronic device, including: a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the data processing method as described in embodiment one.
When being executed by a processor, the computer program realizes the processes of the data processing method embodiments, can achieve the same technical effect, and is not repeated here to avoid repetition.
EXAMPLE five
The embodiment of the present invention further provides a storage medium, where the storage medium includes, but is not limited to, a computer-readable storage medium, and a computer program is stored on the storage medium, and when being executed by a processor, the computer program implements each process of the data processing method embodiment, and can achieve the same technical effect, and is not described herein again to avoid repetition. The storage medium may be a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (11)
1. A method of data processing, the method comprising:
receiving data to be processed, wherein the data to be processed is acquired from a business server by a data acquisition module;
if the data to be processed comprises first format data, converting the first format data into second format data;
performing data processing on the data to be processed to obtain data to be stored;
and storing the data to be stored in a real-time database of a data processing module.
2. The method of claim 1, further comprising, after storing the data to be stored in a real-time database of a data processing module:
receiving a data query request in the real-time database, wherein the data query request is sent to the data processing module by a data query display module;
and sending the query result corresponding to the data query request to the data query display module.
3. The method of claim 1, after receiving the data to be processed, further comprising:
and storing the data to be processed into an offline database of the data processing module.
4. The method of claim 1, wherein if the data to be processed includes data in a first format, converting the data in the first format into data in a second format comprises:
determining a data format in the data to be processed through a grammar engine of the data processing module;
and if the data to be processed comprises first format data, converting the first format data into second format data through the grammar engine.
5. The method of claim 1, wherein performing data processing on the data to be processed to obtain data to be stored comprises:
judging whether the data to be processed accords with a preset rule or not through a rule engine of the data processing module;
under the condition that the data to be processed does not accord with the preset rule, performing data processing on the data to be processed to obtain data to be stored;
and under the condition that the data to be processed accord with the preset rule, determining the data to be processed as the data to be stored.
6. The method of claim 1, wherein storing data to be stored in the real-time database comprises:
and storing the data to be stored into a specified data target in the real-time database according to preset configuration information.
7. The method of claim 1, further comprising:
converting the user behavior event in the service server into third format data through the data acquisition module;
storing the third format data into a cache database of the service server;
and sending the third format data in the cache database to the data processing module every other preset period.
8. A data processing apparatus, characterized in that the apparatus comprises:
the system comprises a receiving unit, a processing unit and a processing unit, wherein the receiving unit is used for receiving data to be processed, and the data to be processed is acquired from a business server by a data acquisition module;
the analysis unit is used for converting the first format data into second format data if the data to be processed comprises the first format data;
the processing unit is used for carrying out data processing on the data to be processed to obtain data to be stored;
and the storage unit is used for storing the data to be stored to a real-time database of the data processing module.
9. A data processing system is characterized in that the system comprises a data acquisition module, a data processing module and a data query display module, wherein,
the data acquisition module is used for sending the data to be processed acquired from the business server to the data processing module;
the data processing module is used for analyzing and processing the data to be processed and then storing the data to be processed into a real-time database of the data processing module;
and the data query display module is used for sending the data query request to the data processing module and displaying the data in the real-time database.
10. An electronic device, comprising: memory, processor and computer program stored on the memory and executable on the processor, which computer program, when being executed by the processor, carries out the steps of the data processing method according to any one of claims 1 to 7.
11. A storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110991637.1A CN113656445A (en) | 2021-08-26 | 2021-08-26 | Data processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110991637.1A CN113656445A (en) | 2021-08-26 | 2021-08-26 | Data processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113656445A true CN113656445A (en) | 2021-11-16 |
Family
ID=78482204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110991637.1A Pending CN113656445A (en) | 2021-08-26 | 2021-08-26 | Data processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113656445A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114462635A (en) * | 2021-12-24 | 2022-05-10 | 南京爱福路汽车科技有限公司 | Service system for providing high QPS statistical query based on vehicle maintenance platform sales data |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
| CN110502662A (en) * | 2019-08-23 | 2019-11-26 | 南京信易达计算技术有限公司 | A kind of Heterogeneous Data Processing system and method |
| CN111198859A (en) * | 2018-11-16 | 2020-05-26 | 北京微播视界科技有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
| CN210724868U (en) * | 2020-03-05 | 2020-06-09 | 中国恩菲工程技术有限公司 | Data processing system and control device |
| CN111274104A (en) * | 2018-11-16 | 2020-06-12 | 北京微播视界科技有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111897808A (en) * | 2020-07-15 | 2020-11-06 | 苏宁金融科技(南京)有限公司 | Data processing method and device, computer equipment and storage medium |
| US20210133212A1 (en) * | 2019-11-04 | 2021-05-06 | Hon Hai Precision Industry Co., Ltd. | Data archiving method and computing device implementing same |
| CN112883095A (en) * | 2021-03-02 | 2021-06-01 | 南京德奈特系统科技有限责任公司 | Method, system, equipment and storage medium for multi-source heterogeneous data convergence |
| CN113138771A (en) * | 2020-01-17 | 2021-07-20 | 北京达佳互联信息技术有限公司 | Data processing method, device, equipment and storage medium |
-
2021
- 2021-08-26 CN CN202110991637.1A patent/CN113656445A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109726074A (en) * | 2018-08-31 | 2019-05-07 | 网联清算有限公司 | Log processing method, device, computer equipment and storage medium |
| CN111198859A (en) * | 2018-11-16 | 2020-05-26 | 北京微播视界科技有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN111274104A (en) * | 2018-11-16 | 2020-06-12 | 北京微播视界科技有限公司 | Data processing method and device, electronic equipment and computer readable storage medium |
| CN110502662A (en) * | 2019-08-23 | 2019-11-26 | 南京信易达计算技术有限公司 | A kind of Heterogeneous Data Processing system and method |
| US20210133212A1 (en) * | 2019-11-04 | 2021-05-06 | Hon Hai Precision Industry Co., Ltd. | Data archiving method and computing device implementing same |
| CN113138771A (en) * | 2020-01-17 | 2021-07-20 | 北京达佳互联信息技术有限公司 | Data processing method, device, equipment and storage medium |
| CN111241182A (en) * | 2020-01-19 | 2020-06-05 | 北京奇艺世纪科技有限公司 | Data processing method and apparatus, storage medium, and electronic apparatus |
| CN210724868U (en) * | 2020-03-05 | 2020-06-09 | 中国恩菲工程技术有限公司 | Data processing system and control device |
| CN111897808A (en) * | 2020-07-15 | 2020-11-06 | 苏宁金融科技(南京)有限公司 | Data processing method and device, computer equipment and storage medium |
| CN112883095A (en) * | 2021-03-02 | 2021-06-01 | 南京德奈特系统科技有限责任公司 | Method, system, equipment and storage medium for multi-source heterogeneous data convergence |
Non-Patent Citations (3)
| Title |
|---|
| 姚世峰: "城市轨道交通云计算应用指南", vol. 2020, 31 December 2020, 北京：中国铁道出版社, pages: 58 - 70 * |
| 张勇、张丽伟: "物联网技术及应用研究", vol. 2020, 30 April 2020, 上海：上海科学技术出版社, pages: 66 - 71 * |
| 韦鹏程、贺方成、黄思行: "基于虚拟化技术的云计算架构的技术与实践探究", vol. 2018, 30 June 2018, 电子科技大学出版社, pages: 199 - 208 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114462635A (en) * | 2021-12-24 | 2022-05-10 | 南京爱福路汽车科技有限公司 | Service system for providing high QPS statistical query based on vehicle maintenance platform sales data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109684352B (en) | Data analysis system, data analysis method, storage medium, and electronic device | |
| US20220398254A1 (en) | Data processing method, platform, computer-readable storage medium and electronic device | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN107908672B (en) | Application report realization method, device and storage medium based on Hadoop platform | |
| CN111885040A (en) | Distributed network situation perception method, system, server and node equipment | |
| CN111324610A (en) | Data synchronization method and device | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN112527886A (en) | Data warehouse system based on urban brain | |
| CN114417408B (en) | Data processing method, device, equipment and storage medium | |
| KR20150118963A (en) | Queue monitoring and visualization | |
| CN112163017B (en) | Knowledge mining system and method | |
| CN112148578A (en) | IT fault defect prediction method based on machine learning | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN115480748A (en) | Service arrangement method, device and storage medium | |
| CN113656445A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN117389700A (en) | Method, device, system and storage medium for processing data of integrated stream and batch | |
| CN114510531A (en) | Database synchronization method and device, electronic equipment and storage medium | |
| CN116303427A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN111352963A (en) | Data statistical method and device | |
| CN116185677A (en) | Automatic fault positioning method, system and medium | |
| CN112286918A (en) | Method and device for fast access conversion of data, electronic equipment and storage medium | |
| CN111949743A (en) | Method, device and equipment for acquiring network operation data | |
| CN111581254A (en) | ETL method and system based on internet financial data | |
| CN111143328A (en) | Agile business intelligent data construction method, system, equipment and storage medium | |
| CN118394772B (en) | Method for updating data asset in real time under change of database table |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |