CN110569301A - Data processing method, system, storage medium and electronic device - Google Patents
Data processing method, system, storage medium and electronic device Download PDFInfo
- Publication number
- CN110569301A CN110569301A CN201810480183.XA CN201810480183A CN110569301A CN 110569301 A CN110569301 A CN 110569301A CN 201810480183 A CN201810480183 A CN 201810480183A CN 110569301 A CN110569301 A CN 110569301A
- Authority
- CN
- China
- Prior art keywords
- data processing
- engine
- rule
- rules
- preset rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Stored Programmes (AREA)
Abstract
The invention provides a data processing method, a data processing system, a storage medium and an electronic device. The method comprises the following steps: providing an interactive interface to display a plurality of preset data processing rules; wherein, each data processing rule is used for realizing a certain step or certain steps in the whole data processing process; the execution of each data processing rule is realized by one or more preset rule engines correspondingly associated with the data processing rule; acquiring one or more data processing rules to be executed through the interactive interface; and calling a preset rule engine associated with the one or more data processing rules to be executed so as to enable the called preset rule engine to execute the corresponding data processing rule, thereby completing the corresponding data processing step. By the data processing method and the system, non-professionals with data processing requirements can easily and efficiently realize data mining and analysis and obtain data processing results.
Description
Technical Field
The present invention relates to the field of data analysis and mining technologies, and in particular, to a data processing method, a data processing system, a storage medium, and an electronic device.
Background
In the big data artificial intelligence era, data gradually becomes one of the most important assets of enterprises and industries, and development decisions and market positioning behaviors are increasingly made based on data analysis. As a competitive resource for constructing data analysis and information processing, the development of big data faces opportunities and challenges brought by a completely new information space. The related data are integrated together, and new information and knowledge can be continuously generated, so that the production rate is improved, and the operation cost is reduced.
However, the current data mining development efficiency is low, the development is complicated, strong professional knowledge is needed, and no unified tool or development mode for simplifying the system exists, so that the data analysis itself is lack of comprehensiveness, and the comprehensive analysis of enterprises is not facilitated. On the basis, enterprises have difficulty in improving the core competitive power and cannot keep or obtain the leading position of the industry for a long time.
disclosure of Invention
In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a data processing method, a system, a storage medium, and an electronic device, which are used to solve the technical problem of the prior art that the development difficulty of data analysis mining is high.
to achieve the above and other related objects, the present invention provides a data processing method, including: providing an interactive interface to display a plurality of preset data processing rules; wherein, each data processing rule is used for realizing a certain step or certain steps in the whole data processing process; the execution of each data processing rule is realized by one or more preset rule engines correspondingly associated with the data processing rule; acquiring one or more data processing rules to be executed through the interactive interface; and calling a preset rule engine associated with the one or more data processing rules to be executed so as to enable the called preset rule engine to execute the corresponding data processing rule, thereby completing the corresponding data processing step.
In an embodiment of the present invention, at least one data processing model is encapsulated in the preset rule engine; each of the data processing models is for implementing one or more data processing operations; and the preset rule engine executes the corresponding data processing rule through the data processing model encapsulated in the preset rule engine so as to realize corresponding data processing operation.
In an embodiment of the present invention, the method further includes: and providing at least one of a viewing service, an export service and a sending service for the data processing result through the interactive interface.
In an embodiment of the present invention, the preset rule engine includes: at least one of a Java engine, Python engine, sql engine, and a stored procedures engine.
To achieve the above and other related objects, the present invention provides a data processing system, comprising: an engine layer and a visualization layer; wherein the engine layer is to: providing a preset rule engine for executing data processing rules; wherein, each data processing rule is used for realizing a certain step or certain steps in the whole data processing process; the execution of each data processing rule is realized by one or more preset rule engines correspondingly associated with the data processing rule; the called preset rule engine executes the corresponding data processing rule, so that the corresponding data processing step is completed; the visualization layer is to: providing an interactive interface to display a plurality of preset data processing rules; and acquiring one or more data processing rules to be executed through the interactive interface, and calling a preset rule engine associated with the one or more data processing rules to be executed.
In an embodiment of the present invention, the system further includes: the data model layer is used for providing at least one data processing model encapsulated in the preset rule engine; each of the data processing models is for implementing one or more data processing operations; and the preset rule engine executes the corresponding data processing rule through the data processing model encapsulated in the preset rule engine so as to realize corresponding data processing operation.
In an embodiment of the present invention, the system further includes: and the data result set layer is used for providing at least one of viewing service, exporting service and sending service for the data processing result through the interactive interface.
In an embodiment of the present invention, the preset rule engine includes: at least one of a Java engine, Python engine, sql engine, and a stored procedures engine.
To achieve the above and other related objects, the present invention provides a storage medium, in which a computer program is stored, which, when loaded and executed by a processor, implements a data processing method as described above.
To achieve the above and other related objects, the present invention provides an electronic device, comprising: a processor, and a memory; wherein the memory is for storing a computer program; the processor is configured to load and execute the computer program to enable the electronic device to execute the data processing method described in any one of the above.
As described above, the data processing method, system, storage medium, and electronic device of the present invention provide a massive data mining analysis platform based on rule description, which facilitates a user to perform a series of data processing operations such as data cleaning, data mining, and data analysis on massive data according to corresponding rules. The complex data processing work is realized through the rule processing engine, the data processing efficiency is improved to the maximum extent, non-professionals with data processing requirements can also easily and efficiently realize data mining and analysis, and data processing results are obtained.
Drawings
Fig. 1 is a flowchart illustrating a data processing method according to an embodiment of the invention.
Fig. 2 is a schematic diagram of an interactive interface according to an embodiment of the invention.
FIG. 3 is a schematic diagram of an interactive interface according to another embodiment of the present invention.
Detailed Description
the embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
Referring to fig. 1, the present embodiment provides a data processing method. Through the data processing method of the embodiment, a user with data processing requirements can execute complex and complicated data processing work and obtain data processing results without developing a data processing model in advance, so that the data processing efficiency is maximized. The data processing method of the embodiment comprises the following steps:
S101: and providing an interactive interface to display a plurality of preset data processing rules.
See fig. 2. Fig. 2 shows a schematic representation of an interactive interface showing 27 schematic data processing rules. The user can clearly learn the name, description, last execution time, execution duration (not shown), last execution status (failure or success), sorting and other information of each data processing rule through the interactive interface, and click the "execute rule" button in the last column to execute the corresponding data processing rule. It should be understood by those skilled in the art that after each data processing rule is executed, the last execution time and the last execution state of each data processing rule are refreshed accordingly, and the sequence number may be set when each data processing rule is established and modified subsequently through user setting.
In this embodiment, each of the data processing rules is used to implement one or more steps in the whole data processing process. In detail, generally, the whole data processing process can be roughly divided into four processes, that is: the method comprises the steps of data cleaning, data integration, data mining and production, and each process can be subdivided into a plurality of steps according to actual conditions. Each data processing rule of the present embodiment is defined for each sub-step or a step composed of several sub-steps, that is, a sub-step is defined as a data processing rule, or a step composed of several sub-steps is defined as a data processing rule. When this data processing rule is executed, the data processing job that the sub-step or the step should complete can be realized. Taking the 27 data processing rules shown in fig. 2 as an example, the execution of each data processing rule is used to implement a step of data processing work, such as: the data processing rule with the number 13 is used for realizing the calculation of the achievement of the knowledge block at the level of the middle test list area when being executed, and the data processing rule with the number 22 is used for realizing the statistics of the general division line number of the five departments at the level of the middle test list school and the like when being executed.
s102: and acquiring one or more data processing rules to be executed through the interactive interface.
Accepting from fig. 2, the user may click the "execute rule" button located in the last column through an input device such as a mouse on the interactive interface to select the data processing rule to be executed.
S103: and calling a preset rule engine associated with the one or more data processing rules to be executed so as to enable the called preset rule engine to execute the corresponding data processing rule, thereby completing the corresponding data processing step.
In this embodiment, the execution of each of the data processing rules is implemented by one or more predetermined rule engines correspondingly associated with the data processing rule. Specifically, the preset rule engine includes: the execution of each data processing rule can be handled by one preset rule engine or a combination of several preset rule engines, and the specific responsibility of each preset rule engine depends on the content of each data processing rule.
For each preset rule engine, at least one data processing model (such as a data mining model and the like) is encapsulated inside the preset rule engine, each data processing model is used for realizing one or more data processing operations, in other words, steps in the data processing rules are realized by the data processing models, and the preset rule engine executes the corresponding data processing rules through the data processing models encapsulated inside the preset rule engine so as to realize the corresponding data processing operations.
In another embodiment, the method further comprises a step of generating and presenting the data processing result, which is also defined by the data processing rule and is executed by the corresponding preset rule engine. At this time, when executing the data processing rule, the preset rule engine generates a Word, Excel or PDF version report according to various pre-stored report templates to present the data processing result. For example, "word generation" as shown in FIG. 3 is a data processing rule for generating a word report, which is responsible for execution by the java engine. And when the java engine executes the data processing rule, generating a word report by using the encapsulated word generation template data model, and then returning to the visual interactive interface.
Furthermore, the interactive interface also provides viewing service, exporting service and sending service for the data processing result, and the result report can be correspondingly displayed, exported and sent or sent to a preset mailbox, a mobile phone and the like when the user clicks.
The present embodiment provides a data processing system. The data processing system is taken as software to be carried on electronic equipment such as a desktop computer, a portable computer, a tablet computer, a smart phone and the like so as to realize the data processing method during operation. The data processing system includes: the visualization layer and the engine layer preferably further comprise: the data model layer, the data result set layer and all the layers are transparent. In the field of computer technology, a feature that is invisible from a certain angle is referred to as transparent, and "transparent" in this sense is the opposite of the transparent sense in social life, for example: the upper layer transparently calls the lower layer interface, namely the upper layer utilizes the service provided by the lower layer, but the realization of the lower layer function is not seen or is not needed. The respective components will be described in detail below.
1. Visualization layer
The visualization layer provides a friendly and concise user input interface, and a user only needs to set a few simple parameters to enable the data processing system to call the engine layer of data analysis according to the keywords set by the user. The engine layer is transparent to the visualization layer.
2. Engine layer
The engine layer provides a common java engine, a python engine, an sql engine, a storage process engine and the like, and can efficiently and quickly execute corresponding data processing rules. The execution of the data processing rules depends on the data processing model of the data model layer. For the engine layer, the data model layer is also transparent, and there is also a loose coupling between the two.
3. Data model layer
The data model layer encapsulates various common data mining models, all model modules are decoupled, and the realization of various models is realized by modularized tuning encapsulation, which is equivalent to a black box. Various new knowledge mined by the layer can be quickly returned to the data result set layer.
4. Data result set layer
And the data result set layer generates personalized Word, Excel and other types of reports according to the templates of various reports. The format and the content of the report are both based on rules, and the report such as word based on the rule description is generated according to the rule description defined by the visualization layer, so that the analysis result is presented. Besides, the report result can be exported or sent to a user mailbox or a mobile phone in a timing mode. The data result set layer not only provides the display of mining results such as visual check reports, but also can provide the functions of downloading and the like.
Those skilled in the art will appreciate that the foregoing division of the data processing system into data layers is merely a logical division and may be fully or partially integrated into one or more physical entities in an actual implementation. And the data layers can be realized in a mode that all software calls through the processing element, can also be realized in a mode that all hardware calls, can also be realized in a mode that part software calls through the processing element, and can be realized in a mode that part hardware calls. For example, the engine layer may be a processing element separately set up, or may be implemented by being integrated in a certain chip, or may be stored in a memory in the form of program code, and a certain processing element calls and executes a function of the engine layer. The other data layers are implemented similarly. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each data layer above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
For example, the above data layers may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), etc. For another example, when the above data layer is implemented in the form of a processing element scheduler code, the processing element may be a general purpose processor, such as: a Central Processing Unit (CPU) or other processor that can call program code. For another example, these data layers may be integrated together and implemented in the form of a system-on-a-chip (SOC).
In addition, the present invention further includes a storage medium and an electronic device, and the technical features in the foregoing embodiments may be applied to the storage medium embodiment and the electronic device embodiment, so that repeated descriptions are omitted.
The storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disk, etc., in which computer programs are stored, which when loaded and executed by a processor, implement all or part of the steps of the data processing method described in the foregoing embodiments.
The electronic device is a device comprising a processor (CPU/MCU/SOC), a memory (ROM/RAM), a communication module (wired/wireless network) and a display module, and is preferably a computer. In particular, the memory stores a computer program, and the processor implements all or part of the steps of the data processing method of the foregoing embodiment when the computer program is loaded and executed.
In summary, the data processing method, the system, the storage medium and the electronic device of the invention are well-defined in hierarchy and convenient to operate, and provide a unified method for analyzing and processing various types of data such as mysql, oracle, java and python for the technical field of big data analysis and mining. The invention processes data analysis and data mining in various fields based on rule description to obtain visual analysis results such as reports of new knowledge. The dependency on the database is simplified through various preset rule engines, the coupling of database operation is reduced, and various modes and ways for operating the database are provided. The method is particularly suitable for non-professionals with data processing requirements, decouples the dependence on various data mining models, effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A data processing method, comprising:
providing an interactive interface to display a plurality of preset data processing rules; wherein, each data processing rule is used for realizing a certain step or certain steps in the whole data processing process; the execution of each data processing rule is realized by one or more preset rule engines correspondingly associated with the data processing rule;
Acquiring one or more data processing rules to be executed through the interactive interface;
And calling a preset rule engine associated with the one or more data processing rules to be executed so as to enable the called preset rule engine to execute the corresponding data processing rule, thereby completing the corresponding data processing step.
2. the method according to claim 1, wherein the preset rules engine is internally encapsulated with at least one data processing model; each of the data processing models is for implementing one or more data processing operations; and the preset rule engine executes the corresponding data processing rule through the data processing model encapsulated in the preset rule engine so as to realize corresponding data processing operation.
3. The method of claim 1, further comprising: and providing at least one of a viewing service, an export service and a sending service for the data processing result through the interactive interface.
4. The method of claim 1, wherein the pre-set rules engine comprises: at least one of a Java engine, Python engine, sql engine, and a stored procedures engine.
5. a data processing system, characterized by comprising at least: an engine layer and a visualization layer; wherein the content of the first and second substances,
The engine layer is used for: providing a preset rule engine for executing data processing rules; wherein, each data processing rule is used for realizing a certain step or certain steps in the whole data processing process; the execution of each data processing rule is realized by one or more preset rule engines correspondingly associated with the data processing rule; the called preset rule engine executes the corresponding data processing rule, so that the corresponding data processing step is completed;
The visualization layer is to: providing an interactive interface to display a plurality of preset data processing rules; and acquiring one or more data processing rules to be executed through the interactive interface, and calling a preset rule engine associated with the one or more data processing rules to be executed.
6. The system of claim 5, further comprising: the data model layer is used for providing at least one data processing model encapsulated in the preset rule engine; each of the data processing models is for implementing one or more data processing operations; and the preset rule engine executes the corresponding data processing rule through the data processing model encapsulated in the preset rule engine so as to realize corresponding data processing operation.
7. The system of claim 5, further comprising: and the data result set layer is used for providing at least one of viewing service, exporting service and sending service for the data processing result through the interactive interface.
8. The system of claim 5, wherein the pre-set rules engine comprises: at least one of a Java engine, Python engine, sql engine, and a stored procedures engine.
9. A storage medium in which a computer program is stored, which, when loaded and executed by a processor, carries out the data processing method according to any one of claims 1 to 4.
10. An electronic device, comprising: a processor, and a memory; wherein the content of the first and second substances,
The memory is used for storing a computer program;
The processor is configured to load and execute the computer program to cause the electronic device to perform the data processing method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810480183.XA CN110569301A (en) | 2018-05-18 | 2018-05-18 | Data processing method, system, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810480183.XA CN110569301A (en) | 2018-05-18 | 2018-05-18 | Data processing method, system, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110569301A true CN110569301A (en) | 2019-12-13 |
Family
ID=68771853
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810480183.XA Pending CN110569301A (en) | 2018-05-18 | 2018-05-18 | Data processing method, system, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110569301A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101201844A (en) * | 2006-12-11 | 2008-06-18 | 国际商业机器公司 | Method and system for discovering data rules |
| CN106777101A (en) * | 2016-12-14 | 2017-05-31 | 深圳天源迪科信息技术股份有限公司 | Data processing engine |
| CN107977441A (en) * | 2017-12-08 | 2018-05-01 | 中国银行股份有限公司 | The method for processing business and transaction processing system of rule-based engine |
-
2018
- 2018-05-18 CN CN201810480183.XA patent/CN110569301A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101201844A (en) * | 2006-12-11 | 2008-06-18 | 国际商业机器公司 | Method and system for discovering data rules |
| CN106777101A (en) * | 2016-12-14 | 2017-05-31 | 深圳天源迪科信息技术股份有限公司 | Data processing engine |
| CN107977441A (en) * | 2017-12-08 | 2018-05-01 | 中国银行股份有限公司 | The method for processing business and transaction processing system of rule-based engine |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10579344B2 (en) | Converting visual diagrams into code | |
| US20210109718A1 (en) | Integration of learning models into a software development system | |
| CN115617327A (en) | Low code page building system, method and computer readable storage medium | |
| CN111061475B (en) | Software code generating method, device, computer equipment and storage medium | |
| CN107273104B (en) | Processing method and device for configuration data structure | |
| US11625248B2 (en) | Operator registration method and apparatus for deep learning framework, device and storage medium | |
| CN108388623B (en) | ER relationship generation method and device, computer equipment and storage medium | |
| CN105512265A (en) | Method and device for displaying data through figure | |
| CN103984626A (en) | Method and device for generating test-case script | |
| US20190392349A1 (en) | Simplified product configuration using table-based rules, rule conflict resolution through voting, and efficient model compilation | |
| WO2016141685A1 (en) | Telecommunications engineering drawing generation method and drawing design client | |
| JP2018520452A (en) | Techniques for constructing generic programs using controls | |
| CN112214210A (en) | Logistics business rule engine and configuration method, device, equipment and storage medium thereof | |
| CN107679056A (en) | Generate method, electronic installation and the readable storage medium storing program for executing of SQL statement | |
| CN111158650B (en) | Report template and report generation method and device | |
| CN110244943B (en) | Graphical service design method, system and software memory based on web and mobile terminal | |
| CN115114146A (en) | Interface test method, device, equipment and storage medium | |
| CN111399816A (en) | Automobile ECU program file generation method and system, storage medium and terminal | |
| CN105468345B (en) | Method for realizing remote change of mobile phone terminal visit operation in quick-elimination system | |
| CN111177562B (en) | Recommendation ordering processing method and device for target object and server | |
| CN113900725A (en) | Interface configuration method and device, computer equipment and storage medium | |
| CN110569301A (en) | Data processing method, system, storage medium and electronic device | |
| CN108845857A (en) | A kind of icon management method and device based on cloud platform | |
| CN103220186A (en) | Communication equipment testing method and system | |
| CN113590660A (en) | Data self-service query method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |