CN109359142B - Data processing method, data processing device, computer equipment and readable storage medium - Google Patents

Data processing method, data processing device, computer equipment and readable storage medium Download PDF

Info

Publication number
CN109359142B
CN109359142B CN201811147551.5A CN201811147551A CN109359142B CN 109359142 B CN109359142 B CN 109359142B CN 201811147551 A CN201811147551 A CN 201811147551A CN 109359142 B CN109359142 B CN 109359142B
Authority
CN
China
Prior art keywords
report
data
elasticsearch
data processing
vue
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201811147551.5A
Other languages
Chinese (zh)
Other versions
CN109359142A (en
Inventor
张毅
孙加光
喻波
王志海
董爱华
安鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Wondersoft Technology Co Ltd
Original Assignee
Beijing Wondersoft Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Wondersoft Technology Co Ltd filed Critical Beijing Wondersoft Technology Co Ltd
Priority to CN201811147551.5A priority Critical patent/CN109359142B/en
Publication of CN109359142A publication Critical patent/CN109359142A/en
Application granted granted Critical
Publication of CN109359142B publication Critical patent/CN109359142B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a report data processing method, a data processing device, computer equipment and a readable storage medium, which can quickly, stably and efficiently display a user-defined report visually through Web under mass data. The report data processing method based on VUE.js, MYSQL and ELASTICSEARCH comprises the steps of synchronously storing report metadata in ELASTICSEARCH; configuring a report page in a VUE.js form; storing the configured report into a MYSQL database; and ELASTICSEARCH is used for processing the report data. The report data processing method meets the requirements of searching and aggregating large data volume, visually displaying data by inquiring an aggregation result and various reports, and various complex and urgent use requirements of various data statistics and data visual display in the data display process.

Description

Data processing method, data processing device, computer equipment and readable storage medium
Technical Field
The invention relates to the technical field of computer data processing, in particular to a report data processing method, a data processing device, computer equipment and a readable storage medium based on VUE.
Background
In order to better understand business data of an enterprise, distribution trends and distribution proportions of various data and the like, managers of the enterprise manage the business data. It is desirable to generate an intuitive data chart so as to be able to intuitively see trends and statistical data of various enterprise activity data, but in the current internet era, the data volume generated by enterprises is almost explosively increased, the capability of the traditional relational database is already in the way, and the requirement cannot be met in mass data storage, query and aggregation, so that the urgent need is to solve the problems of query and aggregation of mass data and visualization display of data through query and aggregation results and various reports. Meanwhile, in order to meet the requirements of various data statistics and data visualization display in the data display process, the displayed report can be flexibly configured and defined according to the requirements. Therefore, how to provide a report configuration method with universality and capable of meeting various display requirements is a problem which needs to be solved urgently at present.
Disclosure of Invention
The invention provides a report data processing method, a data processing device, a computer system and a readable storage medium based on VUE.js, MYSQL and ELASTICSEARCH, which can be flexibly configured and defined according to needs and retrograde movement, and meet the needs of various data statistics and data visualization display in the data display process.
An embodiment of the present invention provides a report data processing method based on vue.js, MYSQL, and ELASTICSEARCH, including:
synchronizing the report metadata stored in ELASTICSEARCH;
configuring a report page in a VUE.js form;
storing the configured report into a MYSQL database;
and ELASTICSEARCH is used for processing the report data. Preferably, the mail self-learning model baseline is a function curve between time and mail safety.
Preferably, the synchronizing the report metadata stored in ELASTICSEARCH specifically includes:
the underlying structure of the data stored in ELASTICSEARCH is synchronized to the database by the metadata synchronization module.
Preferably, the basic structure of the data comprises at least one of: ELASTICSEARCH index name, type name, field type.
Preferably, the configuring the report page in the form of vue.js specifically includes:
accessing a report configuration page, and loading first synchronous metadata to the page to provide basic data for report configuration;
configuring report basic information;
configuring report form screening conditions;
and configuring report aggregation conditions.
Preferably, the processing of the report data by using the ELASTICSEARCH method specifically includes:
reading configured report information from MYSQL;
packaging the screening condition and the aggregation condition of the configured report;
generating a data querier corresponding to the screening condition of the report and a data aggregator corresponding to the aggregation condition;
assembling ELASTICSEARCH query request, request ELASTICSEARCH and data package of the obtained return result according to the data querier and the data aggregation;
packaging the returned result into a data structure meeting the requirement according to the specific configuration information of the report and returning the final result;
and returning the return result to the front end, and displaying the return result by a browser through a VUE.
Another embodiment of the present invention provides a report data processing apparatus based on vue.js, MYSQL, and ELASTICSEARCH, including:
a synchronization unit for synchronizing the report metadata stored in ELASTICSEARCH;
the configuration unit is used for configuring a report page in a VUE.js form;
the storage unit is used for storing the configured report to a MYSQL database;
and the processing unit is used for processing the report data in the ELASTICSEARCH mode.
Preferably, the synchronization unit includes a metadata synchronization module, and is specifically configured to:
the underlying structure of the data stored in ELASTICSEARCH is synchronized to the database by the metadata synchronization module.
Preferably, the basic structure of the data comprises at least one of: ELASTICSEARCH index name, type name, field type.
Preferably, the configuration unit specifically includes:
the loading module is used for accessing the report configuration page and loading the first synchronous metadata to the page to provide basic data for report configuration;
the first configuration module is used for configuring report form basic information;
the second configuration module is used for configuring report screening conditions;
and the third configuration module is used for configuring report aggregation conditions.
Preferably, the processing unit specifically includes:
the reading module is used for reading the configured report information from MYSQL;
the packaging module is used for packaging the screening condition and the aggregation condition of the configured report;
the first generation module is used for generating a data querier corresponding to the screening condition of the report and a data aggregator corresponding to the aggregation condition;
the request module is used for assembling ELASTICSEARCH query requests according to the data querier and the data aggregation, requesting ELASTICSEARCH and carrying out data encapsulation on the obtained return results;
the data return module is used for packaging the return result into a data structure meeting the requirements according to the specific configuration information of the report and returning the final result;
and the second generation module is used for returning the return result to the front end and delivering the return result to the browser to display the return result through a VUE.
Another embodiment of the present invention provides a computer device, which includes a processor, and the processor is configured to implement the report data processing method based on vue.js, MYSQL, and ELASTICSEARCH as described in any one of the above when executing a computer program stored in a memory.
Another embodiment of the present invention provides a computer-readable storage medium having a computer program (instructions) stored thereon, characterized in that: the computer program (instructions), when executed by a processor, implements a report data processing method based on vue.js, MYSQL and ELASTICSEARCH as described in any one of the above.
Aiming at the searching and aggregation of large data volume, the visual display of data through inquiring the aggregation result and various reports, and the statistics of various data and the visual display of data in the data display process, the report data processing method, the data processing device, the computer equipment and the readable storage medium based on the VUE.js, MYSQL and ELASTICSEARCH can quickly, stably and efficiently display the user-defined report visually through Web under mass data.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is an embodiment of a report data processing method based on vue.js, MYSQL and ELASTICSEARCH according to the present invention;
fig. 2 is another embodiment of a report data processing method based on vue.js, MYSQL and ELASTICSEARCH according to the present invention;
FIG. 3 is a model diagram of a report data processing method based on VUE.js, MYSQL and ELASTICSEARCH in the invention;
fig. 4 is an embodiment of a report data processing method based on vue.js, MYSQL and ELASTICSEARCH of the present invention;
FIG. 5 is a computer device embodiment of the present invention;
Detailed Description
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings of the specification, it being understood that the preferred embodiments described herein are merely for illustrating and explaining the present invention, and are not intended to limit the present invention, and that the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.
Based on the defects of the prior art, an embodiment of the present invention provides a report data processing method based on vue.js, MYSQL, and ELASTICSEARCH, where the method includes:
step 101: synchronizing the report metadata stored in ELASTICSEARCH;
step 103: configuring a report page in a VUE.js form;
step 105: storing the configured report into a MYSQL database;
step 107: and ELASTICSEARCH is used for processing the report data.
The embodiment of the invention respectively makes use of: js simple and small core, progressive technology stack, is sufficient to cope with applications of any scale. At the same time Vue focuses only on the view layer and is very easy to learn, very easy to integrate with other libraries or existing items, and responsive data binding and combined view components can be implemented through as simple APIs as possible. Therefore, definition and configuration of the report forms and visual display of the final report forms are realized at the front end by adopting VUE.
MYSQL is used for persisting report definition data and configuration data. Because the configuration of the report and the defined data volume are few, the MYSQL database which is popular, open source and low in cost is adopted.
The elastic search is used for storing mass data generated by an enterprise. The Elasticsearch is inherently distributed, it knows how to manage nodes to provide high expansion and high availability, and therefore Elasticsearch has stable, fast, powerful data query, aggregation capability under large data volumes. The stable, fast and powerful data query and aggregation capabilities are needed for generating various statistical reports, so that the Elasticsearch is adopted for data storage.
For the search and aggregation of a large amount of data, the visual display of data through various reports by querying the aggregation result, and the statistics of various data and the visual display of data in the data display process, the report data processing method, the data processing device, the computer device and the readable storage medium based on vue.js, MYSQL and ELASTICSEARCH provided by this embodiment can be used to quickly, stably and efficiently visually display the user-defined report through the Web under mass data.
In the report data processing method based on vue.js, MYSQL and ELASTICSEARCH in embodiment 1 of the present invention, in step 101, the metadata synchronization module synchronizes to the database through the basic structure of the data stored in ELASTICSEARCH. The main information includes ELASTICSEARCH index name (index in the elastic search), type name (type in the elastic search), field name, field type (field storage type in the elastic search).
It will be apparent to those skilled in the art that the basic structure of the data stored in ELASTICSEARCH includes, in addition to the data structure information described above, other fields and fields that are added as appropriate to the needs of the service.
Compared with the conventional report data processing method, the report data processing method based on VUE.js, MYSQL and ELASTICSEARCH disclosed by the embodiment 1 of the invention has the following technical effects:
1. the report design and configuration of the data needing statistics can be dynamically carried out.
2. Js through VUE, the configuration page is simple, friendly and easy to operate.
3. The specific use can continuously expand the screening condition packaging assembly, the aggregation condition packaging assembly and the return result conversion assembly according to the requirement to enrich the report types supported by the report engine.
As shown in fig. 2, a second embodiment of the present invention provides a report data processing method based on vue.js, MYSQL, and ELASTICSEARCH, including: step 201: the underlying structure of the data stored in ELASTICSEARCH is synchronized to the database by the metadata synchronization module.
It should be noted that the metadata synchronization module synchronizes to the database via the basic structure of the data stored in the elastic search. The main information includes an elastic search index name (index in elastic search), a type name (type in elastic search), a field name, a field type (field storage type in elastic search), and other fields are added according to service requirements.
Step 203: configuring a report page in a VUE.js form;
it should be noted that step 203 in this embodiment 2 specifically includes:
step 2031: accessing a report configuration page in the browser, and loading the metadata synchronized in the first step to the page to provide basic data for report configuration;
step 2032: configuring report basic information:
it should be noted that the report basic information configuration at least includes a report name, a report description, and a report type (column diagram, line diagram, pie chart radar chart, data list, etc.).
Step 2033: report screening condition configuration (optional)
It should be noted that, for those skilled in the art, the screening condition in this embodiment may include at least one of the following configurations, and also includes any combination of the following configurations:
screening fields: providing synchronized metadata in the form of drop-down boxes
Screening field values: if the field in the elasticsearch data is stored as a value that can be combed in traffic to a limited and substantially fixed value, it can be presented by a drop-down box, otherwise an input box type.
And (3) screening types: in keeping with the type of screening provided in the elasticsearch (terms, prefix, wildcard, range, etc.)
Relationship to other screening conditions: the type of relationship between the conditions in the elastic search is kept consistent (with: must; or should; not: mustnot)
Step 2034: configuring report aggregation conditions;
in this embodiment, the polymerization condition main configuration content may include any combination of at least one or more of the following:
screening fields: providing synchronized metadata in the form of drop-down boxes
Polymerization type: consistent with the type of aggregation provided in the elastic search (terms, datehistorgram, count, etc.)
And (3) sub-polymerization: if there are aggregation fields and aggregation types that require recursive configuration of the gametes' aggregations. No configuration is required.
Step 2035: and saving the corresponding report to MYSQL.
In the embodiment of the invention, MySQL is a relational database management system developed by the MySQLAB company in Sweden and belongs to a product under the Oracle flag at present. In terms of WEB applications, MySQL is one of the best RDBMS (Relational Database Management System) application software. The SQL language used by MySQL is the most common standardized language for accessing databases in two. MySQL software has the characteristics of small volume, high speed and low total cost of ownership, and particularly has the characteristic of open source codes, and generally MySQL is selected as a website database for developing small and medium websites.
Step 204: and storing the configured report into a MYSQL database.
It should be noted that step 204 in embodiment 2 of the present invention specifically includes:
step 2041: the report configuration reading component reads the configured report information from MYSQL;
step 2042: the screening condition packaging component packages all screening conditions configured by the report according to the relationship among the screening conditions and combines with Query (Query) api of the elastic search to package the configured screening conditions and generate a corresponding data querier.
And 2043, packaging the configured polymerization conditions by the polymerization condition packaging component according to all polymerization conditions configured by the report and the relationship among the polymerization conditions and by combining with the polymerization (Aggregation) api of the elastic search, and generating a corresponding data aggregator.
Step 2044: and the Es return result packaging component is used for packaging the packaged data querier and the data aggregator into an elastic search query request, requesting the elastic search and simply packaging the obtained return result.
Step 2045: and the return result conversion component encapsulates the return result into a data structure meeting the requirement according to the specific configuration information of the report and returns the final result.
Step 2046: and returning the return result to the front end, and displaying or generating a corresponding report PDF file by other third-party visual plug-ins (Echarts, Antv and the like) through VUE.
It should be noted that, in the embodiment 2 of the present invention, in step 2045, the form and content of the returned result may be selected and set by the developer according to the specific requirements of the target user, and the return result is implemented by using a functional component of a third-party visualization plug-in. The specific form of the returned result is not limited by the description of the embodiment, and the setting can be selected by the user in the actual environment according to the needs of the user.
FIG. 3 is a model schematic diagram of a report data processing method based on VUE.js, MYSQL and ELASTICSEARCH in the invention.
According to the mapping order of the ES to synchronize metadata, it should be noted that, in the ELASTICSEARCH model, when creating an index or using a put mapping Api operation, the mapper module is mainly responsible for adding the definition of the type mapping to the index. And simultaneously supports dynamic mapping types which are not defined in advance. Storing the synchronous metadata into MYSQL, and in the MYSQL data storage, on one hand, reading report configuration from the MYSQL, wherein the report configuration comprises a report configuration reading component, a report screening condition packaging component, a report aggregation condition packaging component, a return result conversion component and the like; reading the metadata on the other hand provides the page with configuration options. And then, storing report configuration information in a database through a browser, wherein the report configuration information comprises data screening conditions, result aggregation fields and aggregation relations of different fields.
And finally, the report engine queries a result from ELASTICSEARCH according to the screening condition and the aggregation condition, and converts the result into a browser or a PDF file for display according to a returned result conversion component.
Aiming at the searching and aggregation of large data volume, the visual display of data through inquiring the aggregation result and various reports, and the statistics of various data and the visual display of data in the data display process, the report data processing method based on VUE.js, MYSQL and ELASTICSEARCH provided by the embodiment of the invention can quickly, stably and efficiently visually display the user-defined report through Web under mass data.
As shown in fig. 4, an embodiment of the present invention provides a report data processing apparatus based on vue.js, MYSQL, and ELASTICSEARCH, including:
a synchronization unit 301 for synchronizing the report metadata stored in ELASTICSEARCH;
it should be noted that, in the embodiment of the present invention, the synchronization unit 301 includes a metadata synchronization module, and is specifically configured to:
the underlying structure of the data stored in ELASTICSEARCH is synchronized to the database by the metadata synchronization module.
The basic structure of the data comprises at least one of the following: ELASTICSEARCH index name, type name, field type.
A configuration unit 303, configured to configure a report page in a vue.js form;
it should be noted that, in the embodiment of the present invention,
the configuration unit 303 includes:
the loading module 3031 is configured to access a report configuration page, and load the first synchronized metadata to the page to provide basic data for report configuration;
the first configuration module 3032 is used for configuring report basic information;
a second configuration module 3033, configured to configure report screening conditions;
and a third configuration module 3034, configured to configure report aggregation conditions.
The storage unit 305 is used for storing the configured report into a MYSQL database;
and the processing unit 307 is used for processing the report data in the manner of ELASTICSEARCH. It should be noted that the mail self-learning model baseline in this embodiment is a function curve between time and mail security, and the mail security is a mail security value associated with a mail feature.
It should be noted that, in this embodiment, the processing unit 307 includes: the reading module 3071 is used for reading the configured report information from the MYSQL;
the packaging module 3072 is used for packaging the screening condition and the aggregation condition of the configured report;
the first generating module 3073 is used for generating a data querier corresponding to the screening condition of the report and a data aggregator corresponding to the aggregation condition;
a request module 3074, configured to assemble ELASTICSEARCH a query request according to the data querier and the data aggregation, request ELASTICSEARCH, and perform data encapsulation on the obtained returned result;
the data returning module 3075 packages the returned result into a data structure meeting the requirements according to the specific configuration information of the report and returns the final result;
and the second generating module 3076 is configured to return the return result to the front end, and deliver the return result to the browser to display the return result through the vue.js set third-party visual plug-in or generate a corresponding report PDF file.
FIG. 5 is a schematic diagram of a computer device provided in an embodiment of the invention. As shown in fig. 4, the computer apparatus 1 includes at least: a processor 20, a memory 30, a computer program 40 (e.g. a control program of a data processing method) stored in said memory 30 and executable on said processor 20.
The computer device 1 may be a computer device with a shooting function, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a server computer, and a personal digital assistant. It will be appreciated by those skilled in the art that the schematic diagram 3 is merely an example of the computer device 1, and does not constitute a limitation of the computer device 1, and may include more or less components than those shown, or combine some components, or different components, for example, the computer device 1 may further include an input and output device, a network access device, a bus, etc.
When the processor 20 executes the computer program 40, the steps of the report data processing method based on vue.js, MYSQL and ELASTICSEARCH described above are implemented, for example, the steps of the report data processing method based on vue.js, MYSQL and ELASTICSEARCH shown in fig. 1, and the steps of the report data processing method based on vue.js, MYSQL and ELASTICSEARCH shown in fig. 2. The processor 20, when executing the computer program 40, implements the functions of the various modules/units in the above-described apparatus embodiments, such as the various units/modules in fig. 4.
Illustratively, the computer program 40 may be partitioned into one or more modules/units that are stored in the memory 30 and executed by the processor 20 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution of the computer program 40 in the terminal 1. For example, the computer program 40 may be divided into the synchronization unit 301, the configuration unit 303, the storage unit 305, the processing unit 307, the loading module 3031, the first configuration module 3032, the second configuration module 3033, the third configuration module 3034, the packaging module 3072, the first generation module 3073, the request module 3074, the data return module 3075, and the second generation module 3076 in fig. 3. For brevity and to avoid repetition, the detailed description of the functions of each unit is omitted.
The Processor 20 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. The general purpose processor may be a microprocessor or said processor 20 may also be any conventional processor or the like, said processor 20 being the step of said vue.js, MYSQL and ELASTICSEARCH based report data processing method/the control center of said computer device, the various parts of the entire vue.js, MYSQL and ELASTICSEARCH based report data processing apparatus/computer device 1 being connected by various interfaces and lines.
The memory 30 is used for storing the computer program 40 and/or the modules/units, and the processor 20 implements various functions of the report data processing apparatus/computer device 1 based on vue.js, MYSQL and ELASTICSEARCH by running or executing the computer program and/or the modules/units stored in the memory 30 and calling the data stored in the memory 30. The memory 30 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data created according to the use of the computer apparatus 1, and the like. In addition, the memory 30 may include a high speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the report data processing method based on vue.js, MYSQL, and ELASTICSEARCH in the foregoing embodiment.
Js, MYSQL and ELASTICSEARCH-based report data processing apparatus// computer device integrated modules/units may be stored in a computer-readable storage medium if implemented in the form of software functional units and sold or used as a stand-alone product. Based on such understanding, all or part of the flow in the method according to the above embodiments may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
In the several embodiments provided in the present invention, it should be understood that the disclosed computer apparatus and method may be implemented in other ways. For example, the above-described embodiments of the computer apparatus are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
In addition, each functional module in each embodiment of the present invention may be integrated into the same processing module, or each module may exist alone physically, or two or more modules may be integrated into the same module. The integrated module can be realized in a hardware form, and can also be realized in a form of hardware and a software functional module.
It will be evident to those skilled in the art that the embodiments of the present invention are not limited to the details of the foregoing illustrative embodiments, and that the embodiments of the present invention are capable of being embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the embodiments being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. Several units, modules or means recited in the system, apparatus or terminal claims may also be implemented by one and the same unit, module or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the embodiments of the present invention and not for limiting, and although the embodiments of the present invention are described in detail with reference to the above preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the embodiments of the present invention without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A report data processing method is characterized by comprising the following steps:
synchronously storing report metadata in a distributed search server;
configuring a report page in a VUE.js form;
storing the configured report forms into a database;
performing distributed processing on the report data;
synchronously storing report metadata in a distributed search server, wherein:
adding the definition of type mapping to the index by a mapper module in the ES, and supporting the dynamic mapping type which is not defined in advance;
the distributed processing of the report data specifically comprises:
reading configured report information from MYSQL;
packaging the screening condition and the aggregation condition of the configured report;
generating a data querier corresponding to the screening condition of the report and a data aggregator corresponding to the aggregation condition;
assembling ELASTICSEARCH a query request according to the data querier and the data aggregator, requesting ELASTICSEARCH and performing data encapsulation on the obtained return result;
packaging the returned result into a data structure meeting the requirement according to the specific configuration information of the report and returning the final result;
and returning the return result to the front end, and displaying the return result by a browser through a VUE.
2. The method of claim 1,
the synchronous storage of the report metadata in the distributed search server specifically includes:
the underlying structure of the data stored in ELASTICSEARCH is synchronized to the database by the metadata synchronization module.
3. The method of claim 2,
the basic structure of the data comprises at least one of the following: ELASTICSEARCH index name, type name, field type.
4. The method of claim 1,
the configuring of the report page in the form of VUE.js specifically comprises:
accessing a report configuration page, and loading first synchronous metadata to the page to provide basic data for report configuration; configuring report basic information;
configuring report form screening conditions;
and configuring report aggregation conditions.
5. A report data processing apparatus, comprising:
the synchronization unit is used for synchronously storing report metadata in the distributed search server;
the configuration unit is used for configuring a report page in a VUE.js form;
the storage unit is used for storing the configured report forms into a database;
the processing unit is used for performing distributed processing on the report data;
the synchronization unit: adding the definition of type mapping to the index by a mapper module in the ES, and supporting the dynamic mapping type which is not defined in advance;
the processing unit specifically includes:
the reading submodule is used for reading the configured report information from MYSQL;
the packaging submodule is used for packaging the screening condition and the aggregation condition of the configured report;
the aggregation sub-module is used for generating a data querier corresponding to the screening condition of the report and a data aggregator corresponding to the aggregation condition;
the second query submodule is used for assembling ELASTICSEARCH query requests according to the data querier and the data aggregator, requesting ELASTICSEARCH and performing data encapsulation on obtained return results;
the obtaining result submodule is used for packaging the returned result into a data structure meeting the requirement according to the specific configuration information of the report and returning the final result;
and the report display submodule is used for returning the return result to the front end and delivering the return result to the browser to display the return result through a VUE.
6. The reporting data processing apparatus as in claim 5,
the synchronization unit includes a metadata synchronization module, and is specifically configured to:
the underlying structure of the data stored in ELASTICSEARCH is synchronized to the database by the metadata synchronization module.
7. The data processing apparatus of claim 6,
the basic structure of the data comprises at least one of the following: ELASTICSEARCH index name, type name, field type.
8. A computer device, characterized in that the computer device comprises a processor for implementing the report data processing method according to any of claims 1-4 when executing the computer program stored in the memory.
9. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when being executed by a processor, realizes the steps of the report data processing method according to any of claims 1-4.
CN201811147551.5A 2018-09-29 2018-09-29 Data processing method, data processing device, computer equipment and readable storage medium Active CN109359142B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811147551.5A CN109359142B (en) 2018-09-29 2018-09-29 Data processing method, data processing device, computer equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811147551.5A CN109359142B (en) 2018-09-29 2018-09-29 Data processing method, data processing device, computer equipment and readable storage medium

Publications (2)

Publication Number Publication Date
CN109359142A CN109359142A (en) 2019-02-19
CN109359142B true CN109359142B (en) 2020-11-27

Family

ID=65348070

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811147551.5A Active CN109359142B (en) 2018-09-29 2018-09-29 Data processing method, data processing device, computer equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN109359142B (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110688423A (en) * 2019-09-06 2020-01-14 中国平安财产保险股份有限公司 Data processing and displaying method, system, equipment and storage medium
CN111914192A (en) * 2020-07-08 2020-11-10 新奥数能科技有限公司 Method and device for displaying equipment data
CN112445464A (en) * 2020-11-27 2021-03-05 智业软件股份有限公司 Data statistics and display method, terminal equipment and storage medium
CN112861484B (en) * 2021-02-20 2023-03-14 山东旗帜信息有限公司 Method, equipment and storage medium for editing report form through headless browser
CN113032436B (en) * 2021-04-16 2022-05-31 苏州臻璇数据信息技术有限公司 Searching method and device based on article content and title

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107577515A (en) * 2017-10-10 2018-01-12 北京小度信息科技有限公司 Extended method, device, electronic equipment and the storage medium of visualization component
CN107861859A (en) * 2017-11-22 2018-03-30 北京汇通金财信息科技有限公司 A kind of blog management method and system based on micro services framework
CN107958080A (en) * 2017-12-14 2018-04-24 上海特易信息科技有限公司 A kind of big data report processing method based on ElasticSearch
CN108170654A (en) * 2017-12-22 2018-06-15 北京鑫丰南格科技股份有限公司 A kind of electronic forms editing device and its development approach
CN108509326A (en) * 2018-04-09 2018-09-07 四川长虹电器股份有限公司 A kind of service state statistical method and system based on nginx daily records
CN108519967A (en) * 2018-04-08 2018-09-11 深圳乐信软件技术有限公司 Chart method for visualizing, device, terminal and storage medium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10200390B2 (en) * 2016-02-29 2019-02-05 Palo Alto Networks, Inc. Automatically determining whether malware samples are similar

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107577515A (en) * 2017-10-10 2018-01-12 北京小度信息科技有限公司 Extended method, device, electronic equipment and the storage medium of visualization component
CN107861859A (en) * 2017-11-22 2018-03-30 北京汇通金财信息科技有限公司 A kind of blog management method and system based on micro services framework
CN107958080A (en) * 2017-12-14 2018-04-24 上海特易信息科技有限公司 A kind of big data report processing method based on ElasticSearch
CN108170654A (en) * 2017-12-22 2018-06-15 北京鑫丰南格科技股份有限公司 A kind of electronic forms editing device and its development approach
CN108519967A (en) * 2018-04-08 2018-09-11 深圳乐信软件技术有限公司 Chart method for visualizing, device, terminal and storage medium
CN108509326A (en) * 2018-04-09 2018-09-07 四川长虹电器股份有限公司 A kind of service state statistical method and system based on nginx daily records

Also Published As

Publication number Publication date
CN109359142A (en) 2019-02-19

Similar Documents

Publication Publication Date Title
CN109359142B (en) Data processing method, data processing device, computer equipment and readable storage medium
US8874600B2 (en) System and method for building a cloud aware massive data analytics solution background
JP6535031B2 (en) Data query method and apparatus
US9418101B2 (en) Query optimization
US10169437B2 (en) Triplestore replicator
US9817877B2 (en) Optimizing data processing using dynamic schemas
US9594828B2 (en) Executing structured queries on text records of unstructured data
US8977623B2 (en) Method and system for search engine indexing and searching using the index
US8924373B2 (en) Query plans with parameter markers in place of object identifiers
CN110162512B (en) Log retrieval method, device and storage medium
CN102999574A (en) Positioning of relative content item via crossing plural different content sources
CN110427438A (en) Data processing method and its device, electronic equipment and medium
CN108363741B (en) Big data unified interface method, device, equipment and storage medium
US8095532B2 (en) Apparatus and method for generating report data in a multi-user environment
Corradi et al. Linked data for open government: The case of Bologna
US20150169675A1 (en) Data access using virtual retrieve transformation nodes
US9230022B1 (en) Customizable result sets for application program interfaces
CN113254819B (en) Page rendering method, system, equipment and storage medium
US11609924B2 (en) Database query execution on multiple databases
CN109543079B (en) Data query method and device, computing equipment and storage medium
Kondylakis et al. Enabling joins over cassandra NoSQL databases
US20170344549A1 (en) Enhanced database query processing
CN113761053A (en) Data query method and device, electronic equipment and storage medium
CN109582718B (en) Data processing method, device and storage medium
CN112597185A (en) Big data query method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant