CN111046083A - Data analysis method and system and big data platform - Google Patents
Data analysis method and system and big data platform Download PDFInfo
- Publication number
- CN111046083A CN111046083A CN201911284301.0A CN201911284301A CN111046083A CN 111046083 A CN111046083 A CN 111046083A CN 201911284301 A CN201911284301 A CN 201911284301A CN 111046083 A CN111046083 A CN 111046083A
- Authority
- CN
- China
- Prior art keywords
- data
- service data
- target service
- analysis request
- basic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 128
- 238000000034 method Methods 0.000 title claims abstract description 54
- 238000007619 statistical method Methods 0.000 claims abstract description 37
- 238000012545 processing Methods 0.000 claims abstract description 20
- 238000004458 analytical method Methods 0.000 claims description 27
- 230000002452 interceptive effect Effects 0.000 claims description 18
- 230000000007 visual effect Effects 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 5
- 238000009877 rendering Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 13
- 230000003993 interaction Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 6
- 230000009471 action Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000008447 perception Effects 0.000 description 2
- 238000007639 printing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Probability & Statistics with Applications (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a data analysis method, a data analysis system and a big data platform, wherein the method is applied to the big data platform, the big data platform synchronously stores basic data of at least one service system, and the method comprises the following steps: acquiring a service data analysis request; analyzing the service data analysis request, and determining the identifier and the index item of the target service data; and extracting the target service data from the pre-stored basic data according to the identification of the target service data, and performing statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data. The whole data analysis process of the invention does not need to occupy the computing resources of the service system, does not influence the operation of the service system, realizes the data analysis of the service data of at least one service system, and simultaneously utilizes the powerful function of a big data platform to improve the data analysis efficiency and the processing capacity.
Description
Technical Field
The invention relates to the technical field of data analysis, in particular to a data analysis method, a data analysis system and a big data platform.
Background
The data analysis refers to a process of analyzing a large amount of collected data by using a proper statistical analysis method, extracting useful information, forming a conclusion, and further performing detailed research and summary on the data. In practical applications, data analysis is an important technical means to help people make judgments so as to take appropriate actions.
When analyzing the service data, the service system directly analyzes the related service data, and the service data analysis operation occupies the computing resources of the service system and affects the operation performance of the service system.
Disclosure of Invention
In view of this, the present invention provides a data analysis method, system and big data platform, which analyze the service data on the big data platform without affecting the operation of the service system.
In order to achieve the above purpose, the invention provides the following specific technical scheme:
a data analysis method is applied to a big data platform, the big data platform synchronously stores basic data of at least one business system, and the method comprises the following steps:
acquiring a service data analysis request;
analyzing the service data analysis request, and determining the identifier and the index item of the target service data;
and extracting the target service data from pre-stored basic data according to the identification of the target service data, and performing statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
Optionally, the obtaining of the service data analysis request includes:
and receiving the service data analysis request sent by a user through a visual interactive interface of the big data platform.
Optionally, the obtaining of the service data analysis request includes:
and receiving the service data analysis request sent by the service system, wherein the service data analysis request comprises the identifier of the target service data added by the service system after receiving the analysis request sent by the user, and the analysis request sent by the user carries the index item.
Optionally, the extracting the target service data from the pre-stored basic data according to the identifier of the target service data includes:
reading pre-stored authority configuration information;
judging whether a user sending the service data analysis request has access authority to the target service data or not according to the authority configuration information;
and if the access right to the target service data exists, extracting the target service data from pre-stored basic data according to the identification of the target service data.
Optionally, the method further includes:
synchronously extracting newly added original data from a core database of the at least one service system;
preprocessing the original data to obtain the basic data;
and storing the basic data.
Optionally, the step of synchronously storing basic data of at least one service system by the big data platform includes:
acquiring an identifier of basic data of a service system;
determining a basic subject domain of basic data of the business system according to the identification of the basic data of the business system;
and storing the basic data of the business system in a storage space corresponding to the basic subject domain to which the business system belongs.
Optionally, after obtaining the statistical analysis result of the target service data, the method further includes:
generating a report in a preset format according to the statistical analysis result of the target service data;
storing the generated report in a cache;
rendering the report in the cache;
and displaying the rendered report on a visual interactive interface of the big data platform.
A big data platform, comprising:
the data layer is used for synchronously storing basic data of at least one service system;
the interactive interface layer is used for acquiring a service data analysis request sent by a user and forwarding the service data analysis request to the management control layer;
the management control layer is used for analyzing the service data analysis request, determining the identifier and the index item of target service data, and sending the identifier and the index item of the target service data to the data processing layer;
and the data processing layer is used for extracting the target service data from the data layer according to the identification of the target service data and carrying out statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
Optionally, the interactive interface layer is specifically configured to receive the service data analysis request sent by the user through the visual interactive interface of the big data platform.
Optionally, the interaction interface layer is specifically configured to receive the service data analysis request sent by the service system, where the service data analysis request includes an identifier of the target service data added by the service system after receiving an analysis request sent by a user, and the analysis request sent by the user carries the index item.
Optionally, the management control layer is specifically configured to:
reading the authority configuration information stored in the data layer;
judging whether a user sending the service data analysis request has access authority to the target service data or not according to the authority configuration information;
and if the access right to the target service data exists, extracting the target service data from pre-stored basic data according to the identification of the target service data.
Optionally, the data layer is further configured to:
synchronously extracting newly added original data from a core database of the at least one service system;
preprocessing the original data to obtain the basic data;
and storing the basic data.
Optionally, the data layer is specifically configured to:
acquiring an identifier of basic data of a service system;
determining a basic subject domain of basic data of the business system according to the identification of the basic data of the business system;
and storing the basic data of the business system in a storage space corresponding to the basic subject domain to which the business system belongs.
Optionally, the big data platform further includes a cache mechanism layer;
the data processing layer is further used for generating a report in a preset format according to the statistical analysis result of the target service data, and storing the generated report in the cache of the cache mechanism layer;
the management control layer is also used for sending the report in the cache to the interaction interface layer;
and the interactive interface layer is also used for rendering the received report and displaying the rendered report on a visual interactive interface of the big data platform.
A data analysis system, comprising: the big data platform and at least one business system;
the big data platform is in communication connection with the service system and synchronously stores basic data of the service system;
the big data platform is used for executing the data analysis method.
Optionally, the service system is configured to:
receiving an analysis request which is sent by a user and carries index items;
adding an identifier of target service data in the analysis request according to the index item to obtain a service data analysis request;
sending the service data analysis request to the big data platform;
and receiving and displaying the statistical analysis result of the target service data fed back by the big data platform.
Compared with the prior art, the invention has the following beneficial effects:
the data analysis method disclosed by the invention is applied to a big data platform independent of a business system, and the big data platform enables the big data platform to be consistent with basic data in the business system by synchronously storing the basic data of at least one business system, so that the accuracy of data analysis is not influenced. After receiving the service data analysis request, the target service data corresponding to the service data analysis request can be extracted from the pre-stored basic data, and the target service data is subjected to statistical processing, so that the whole data analysis process does not occupy the computing resources of the service system, the operation of the service system is not influenced, the data analysis of the service data of at least one service system is realized, and meanwhile, the strong processing capacity of a large data platform is utilized, and the data analysis efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic flow chart of a data analysis method according to an embodiment of the present invention;
FIG. 2 is a schematic structural diagram of a big data platform according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of another big data platform disclosed in the embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data analysis system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The inventor finds the following problems in the existing data analysis method through research:
1. when analyzing the service data, the service system directly analyzes the related service data, and the service data analysis operation occupies the computing resources of the service system and affects the operation performance of the service system.
2. With the rapid development of information technology, the types of business systems in enterprises are more and more, the analysis of business data tends to be refined and complicated, and the situation that the business data related to a plurality of business systems needs to be comprehensively analyzed often occurs, but the current business systems can only analyze the business data in the system, and cannot comprehensively analyze the business data of the business systems.
In order to solve the technical problems, the invention provides a data analysis method which is applied to a big data platform independent of a business system, the whole data analysis process does not need to occupy the computing resources of the business system and does not influence the operation of the business system, the big data platform synchronously stores the basic data of at least one business system, so that the big data platform is consistent with the basic data in the business system, the data analysis of the business data of at least one business system is realized, and meanwhile, the strong processing capacity of the big data platform is utilized, and the data analysis efficiency is improved.
Specifically, referring to fig. 1, the data analysis method disclosed in the embodiment of the present invention includes the following steps:
s101: acquiring a service data analysis request;
there may be various methods for obtaining the service data analysis request.
One of the methods is as follows: and receiving a service data analysis request sent by a user through a visual interactive interface of the big data platform, namely the user can directly interact with the big data platform.
The other mode is as follows: in this way, a user can also follow the previous data analysis process and directly send an analysis request carrying an index item to the service system, and after receiving the analysis request, the service system adds an identifier of target service data in the analysis request to obtain a service data analysis request and sends the service data analysis request to the big data platform. In this way, the user can use the previous data analysis process without perception, the purpose of service data analysis can be achieved, and the user experience is improved on the basis of not influencing the performance of the service system.
S102: analyzing the service data analysis request, and determining the identifier and the index item of the target service data;
the identifier of the target service data indicates the range of the target data, and the index item indicates an index item which needs to be obtained after statistical analysis.
S103: and extracting the target service data from pre-stored basic data according to the identification of the target service data, and performing statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
It should be noted that the big data platform synchronously stores basic data of at least one business system, the basic data is not all data of the business system, but data which may be used in subsequent analysis, and the basic data is data extracted from a core database of the business system according to a preset rule.
For the convenience of subsequent analysis, the big data platform may synchronously extract newly added original data from the core database of at least one service system, preprocess the original data, such as cleaning, processing, etc., to obtain basic data, and store the basic data.
In order to store basic data in a big data platform, a corresponding storage space is set for each basic subject domain in advance, for example, a data warehouse, when the basic data of a business system is stored, firstly, an identifier of the basic data of the business system is obtained, then, a basic subject domain to which the basic data of the business system belongs is determined according to the identifier of the basic data of the business system, and finally, the basic data of the business system is stored in the storage space corresponding to the basic subject domain to which the basic data of the business system belongs.
The underlying subject area represents the underlying data range, such as deposit, credit, settlement, funds, finance, etc.
On the basis, when data analysis is carried out, the target business data can be directly extracted from the storage space of the basic subject domain corresponding to the identification of the target business data, and the target business data is subjected to statistical analysis according to the index items to obtain the statistical analysis result of the target business data.
When the target service data volume is large, in order to improve the processing efficiency, the analysis and statistics task of the target service data can be split into a plurality of subtasks, and the subtasks are processed in parallel in a multithreading manner, so that the data statistics and analysis efficiency is improved.
After the statistical analysis result of the target service data is obtained, a report can be generated in a preset format according to the statistical analysis result of the target service data, the preset format is such as HTML, Excel and the like, and the generated report is stored in a cache so as to facilitate subsequent checking.
And when the report needs to be displayed to the user, rendering the report in the cache, and displaying the rendered report on a visual interactive interface of the big data platform.
That is to say, the user can carry out report inquiry, report display, report export and report printing through the visual interactive interface of the big data platform.
Because the big data platform can realize data analysis on the service data of at least one service system, that is, a plurality of service systems can carry out data analysis through the big data platform, in order to ensure the security of data, the users of each service system and the upper layer users need to be assigned with different authorities, and only have the authority to access corresponding data.
On the basis, permission configuration information is stored in the big data platform in advance, and the permission configuration information comprises the corresponding permission of each user.
After receiving a service data analysis request, analyzing the service data analysis request, determining an identifier and an index item of target service data, reading pre-stored authority configuration information, and judging whether a user sending the service data analysis request has access authority to the target service data according to the authority configuration information.
If the user has the access right to the target service data, extracting the target service data from the pre-stored basic data according to the identification of the target service data, performing statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data, and further displaying the statistical analysis result to the user.
If the user does not have the access right to the target service data, prompting that the user has no access right or prompting that a corresponding statistical analysis result cannot be displayed.
It can be seen that the data analysis method disclosed in this embodiment is applied to a big data platform independent of a service system, and the big data platform enables the big data platform to be consistent with basic data in the service system by synchronously storing the basic data of at least one service system, without affecting the accuracy of data analysis. After receiving the service data analysis request, the target service data corresponding to the service data analysis request can be extracted from the pre-stored basic data, and the target service data is subjected to statistical processing, so that the whole data analysis process does not occupy the computing resources of the service system, the operation of the service system is not influenced, the data analysis of the service data of at least one service system is realized, and meanwhile, the strong processing capacity of a large data platform is utilized, and the data analysis efficiency is improved.
Based on the data analysis method disclosed in the foregoing embodiment, this embodiment correspondingly discloses a big data platform, please refer to fig. 2, where the big data platform includes:
a data layer 100 for synchronously storing basic data of at least one service system;
the data layer can store basic data through databases such as Oracle and HBASE, and can also store the basic data through a predefined data warehouse model, for example, each basic subject domain corresponds to a storage space in the data warehouse model.
The interactive interface layer 200 is configured to obtain a service data analysis request sent by a user, and forward the service data analysis request to a management control layer;
the interaction interface layer can realize interaction between a user and the big data platform, such as report query, report display, report export, report printing and the like.
The management control layer 300 is configured to analyze the service data analysis request, determine an identifier and an index item of target service data, and send the identifier and the index item of the target service data to a data processing layer;
the management control layer can realize a timing task function, such as a timing execution of a preset data analysis task, an entitlement control function, a fault tolerance processing function, a personalized support function, a safety control function and a response mechanism function.
The data processing layer 400 is configured to extract the target service data from the data layer according to the identifier of the target service data, and perform statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
Optionally, the interaction interface layer 200 is specifically configured to receive the service data analysis request sent by the user through the visual interaction interface of the big data platform.
Optionally, the interaction interface layer 200 is specifically configured to receive the service data analysis request sent by the service system, where the service data analysis request includes an identifier of the target service data added by the service system after receiving an analysis request sent by a user, and the analysis request sent by the user carries the index item.
Optionally, the management control layer 300 is specifically configured to:
reading the authority configuration information stored in the data layer;
judging whether a user sending the service data analysis request has access authority to the target service data or not according to the authority configuration information;
and if the access right to the target service data exists, extracting the target service data from pre-stored basic data according to the identification of the target service data.
Optionally, the data layer 100 is further configured to:
synchronously extracting newly added original data from a core database of the at least one service system;
preprocessing the original data to obtain the basic data;
and storing the basic data.
Optionally, the data layer 100 is specifically configured to:
acquiring an identifier of basic data of a service system;
determining a basic subject domain of basic data of the business system according to the identification of the basic data of the business system;
and storing the basic data of the business system in a storage space corresponding to the basic subject domain to which the business system belongs.
Optionally, referring to fig. 3, the big data platform further includes a cache mechanism 500;
the data processing layer 400 is further configured to generate a report in a preset format according to the statistical analysis result of the target service data, and store the generated report in the cache of the cache mechanism layer;
the management control layer 300 is further configured to send the report in the cache to the interaction interface layer;
the interactive interface layer 200 is further configured to render the received report, and display the rendered report on the visual interactive interface of the big data platform.
The cache mechanism layer 500 can implement Redis caching of the report, record logs and facilitate subsequent tracing.
According to the big data platform disclosed by the embodiment, the big data platform synchronously stores the basic data of at least one service system, so that the basic data of the big data platform and the basic data of the service system are consistent, and the accuracy of data analysis is not influenced. After receiving the service data analysis request, the target service data corresponding to the service data analysis request can be extracted from the pre-stored basic data, and the target service data is subjected to statistical processing, so that the whole data analysis process does not occupy the computing resources of the service system, the operation of the service system is not influenced, the data analysis of the service data of at least one service system is realized, and meanwhile, the strong processing capacity of a large data platform is utilized, and the data analysis efficiency is improved.
The embodiment of the present invention further discloses a data analysis system, please refer to fig. 4, which includes: the big data platform and at least one business system;
the big data platform is in communication connection with the service system and synchronously stores basic data of the service system;
the big data platform is used for executing the following data analysis method:
acquiring a service data analysis request;
analyzing the service data analysis request, and determining the identifier and the index item of the target service data;
and extracting the target service data from pre-stored basic data according to the identification of the target service data, and performing statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
Further, the acquiring of the service data analysis request includes:
and receiving the service data analysis request sent by a user through a visual interactive interface of the big data platform.
Further, the acquiring of the service data analysis request includes:
and receiving the service data analysis request sent by the service system, wherein the service data analysis request comprises the identifier of the target service data added by the service system after receiving the analysis request sent by the user, and the analysis request sent by the user carries the index item.
Further, the extracting the target service data from the pre-stored basic data according to the identifier of the target service data includes:
reading pre-stored authority configuration information;
judging whether a user sending the service data analysis request has access authority to the target service data or not according to the authority configuration information;
and if the access right to the target service data exists, extracting the target service data from pre-stored basic data according to the identification of the target service data.
Further, the method further comprises:
synchronously extracting newly added original data from a core database of the at least one service system;
preprocessing the original data to obtain the basic data;
and storing the basic data.
Further, the big data platform synchronously stores basic data of at least one business system, and comprises:
acquiring an identifier of basic data of a service system;
determining a basic subject domain of basic data of the business system according to the identification of the basic data of the business system;
and storing the basic data of the business system in a storage space corresponding to the basic subject domain to which the business system belongs.
Further, after the obtaining of the result of the statistical analysis of the target service data, the method further includes:
generating a report in a preset format according to the statistical analysis result of the target service data;
storing the generated report in a cache;
rendering the report in the cache;
and displaying the rendered report on a visual interactive interface of the big data platform.
Optionally, the service system is configured to:
receiving an analysis request which is sent by a user and carries index items;
adding an identifier of target service data in the analysis request according to the index item to obtain a service data analysis request;
sending the service data analysis request to the big data platform;
and receiving and displaying the statistical analysis result of the target service data fed back by the big data platform.
That is to say, the user can also follow the previous data analysis flow and directly send an analysis request to the service system, and after receiving the analysis request, the service system adds the identifier of the target service data to the analysis request to obtain a service data analysis request, and sends the service data analysis request to the big data platform. In this way, the user can use the previous data analysis process without perception, the purpose of service data analysis can be achieved, and the user experience is improved on the basis of not influencing the performance of the service system.
It can be seen that, in the data analysis system disclosed in this embodiment, data analysis is performed on a big data platform, the whole data analysis process does not need to occupy computing resources of a service system, operation of the service system is not affected, data analysis on service data of at least one service system is realized, and meanwhile, the strong processing capability of the big data platform is utilized, so that the data analysis efficiency is improved.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data analysis method is applied to a big data platform, the big data platform synchronously stores basic data of at least one business system, and the method comprises the following steps:
acquiring a service data analysis request;
analyzing the service data analysis request, and determining the identifier and the index item of the target service data;
and extracting the target service data from pre-stored basic data according to the identification of the target service data, and performing statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
2. The method of claim 1, wherein the obtaining the service data analysis request comprises:
and receiving the service data analysis request sent by a user through a visual interactive interface of the big data platform.
3. The method of claim 1, wherein the obtaining the service data analysis request comprises:
and receiving the service data analysis request sent by the service system, wherein the service data analysis request comprises the identifier of the target service data added by the service system after receiving the analysis request sent by the user, and the analysis request sent by the user carries the index item.
4. The method according to claim 2 or 3, wherein the extracting the target service data from the pre-stored basic data according to the identification of the target service data comprises:
reading pre-stored authority configuration information;
judging whether a user sending the service data analysis request has access authority to the target service data or not according to the authority configuration information;
and if the access right to the target service data exists, extracting the target service data from pre-stored basic data according to the identification of the target service data.
5. The method of claim 1, further comprising:
synchronously extracting newly added original data from a core database of the at least one service system;
preprocessing the original data to obtain the basic data;
and storing the basic data.
6. The method of claim 1 or 5, wherein the big data platform synchronously stores basic data of at least one business system, and comprises:
acquiring an identifier of basic data of a service system;
determining a basic subject domain of basic data of the business system according to the identification of the basic data of the business system;
and storing the basic data of the business system in a storage space corresponding to the basic subject domain to which the business system belongs.
7. The method of claim 1, wherein after obtaining the result of the statistical analysis of the target traffic data, the method further comprises:
generating a report in a preset format according to the statistical analysis result of the target service data;
storing the generated report in a cache;
rendering the report in the cache;
and displaying the rendered report on a visual interactive interface of the big data platform.
8. A big data platform, comprising:
the data layer is used for synchronously storing basic data of at least one service system;
the interactive interface layer is used for acquiring a service data analysis request sent by a user and forwarding the service data analysis request to the management control layer;
the management control layer is used for analyzing the service data analysis request, determining the identifier and the index item of target service data, and sending the identifier and the index item of the target service data to the data processing layer;
and the data processing layer is used for extracting the target service data from the data layer according to the identification of the target service data and carrying out statistical analysis on the target service data according to the index item to obtain a statistical analysis result of the target service data.
9. A data analysis system, comprising: the big data platform and at least one business system;
the big data platform is in communication connection with the service system and synchronously stores basic data of the service system;
the big data platform is used for executing the data analysis method of any one of claims 1 to 7.
10. The data analysis system of claim 9, wherein the business system is configured to:
receiving an analysis request which is sent by a user and carries index items;
adding an identifier of target service data in the analysis request according to the index item to obtain a service data analysis request;
sending the service data analysis request to the big data platform;
and receiving and displaying the statistical analysis result of the target service data fed back by the big data platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911284301.0A CN111046083A (en) | 2019-12-13 | 2019-12-13 | Data analysis method and system and big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911284301.0A CN111046083A (en) | 2019-12-13 | 2019-12-13 | Data analysis method and system and big data platform |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111046083A true CN111046083A (en) | 2020-04-21 |
Family
ID=70236204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911284301.0A Pending CN111046083A (en) | 2019-12-13 | 2019-12-13 | Data analysis method and system and big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111046083A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111652485A (en) * | 2020-05-25 | 2020-09-11 | 青海绿能数据有限公司 | New energy data acquisition and analysis system based on big data platform |
| CN112269830A (en) * | 2020-10-20 | 2021-01-26 | 苏州莱锦机电自动化有限公司 | Big data analysis method, system, computer equipment and storage medium thereof |
| CN112527848A (en) * | 2020-12-22 | 2021-03-19 | 苏州科达科技股份有限公司 | Multi-data-source-based report data query method, device, system and storage medium |
| CN113836216A (en) * | 2021-09-06 | 2021-12-24 | 飞算数智科技(深圳)有限公司 | Data display method and device, storage medium and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140032263A1 (en) * | 2012-07-25 | 2014-01-30 | Indix Corporation | Adaptive gathering of structured and unstructured data system and method |
| CN104394211A (en) * | 2014-11-21 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Hadoop-based user behavior analysis system design and implementation method |
| CN106355489A (en) * | 2016-08-26 | 2017-01-25 | 山东尚捷信息科技有限公司 | Data center system and data processing method for management |
| CN108696531A (en) * | 2018-06-08 | 2018-10-23 | 武汉思普崚技术有限公司 | A kind of security strategy adaptive analysis and big data Visualization Platform system |
| CN109408548A (en) * | 2018-09-19 | 2019-03-01 | 中国电力科学研究院有限公司 | A kind of urban electric power big data application system and method |
-
2019
- 2019-12-13 CN CN201911284301.0A patent/CN111046083A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140032263A1 (en) * | 2012-07-25 | 2014-01-30 | Indix Corporation | Adaptive gathering of structured and unstructured data system and method |
| CN104394211A (en) * | 2014-11-21 | 2015-03-04 | 浪潮电子信息产业股份有限公司 | Hadoop-based user behavior analysis system design and implementation method |
| CN106355489A (en) * | 2016-08-26 | 2017-01-25 | 山东尚捷信息科技有限公司 | Data center system and data processing method for management |
| CN108696531A (en) * | 2018-06-08 | 2018-10-23 | 武汉思普崚技术有限公司 | A kind of security strategy adaptive analysis and big data Visualization Platform system |
| CN109408548A (en) * | 2018-09-19 | 2019-03-01 | 中国电力科学研究院有限公司 | A kind of urban electric power big data application system and method |
Non-Patent Citations (1)
| Title |
|---|
| 江樱等: "基于大数据技术的电力全业务数据运营管理平台研究", 《自动化技术与应用》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111652485A (en) * | 2020-05-25 | 2020-09-11 | 青海绿能数据有限公司 | New energy data acquisition and analysis system based on big data platform |
| CN112269830A (en) * | 2020-10-20 | 2021-01-26 | 苏州莱锦机电自动化有限公司 | Big data analysis method, system, computer equipment and storage medium thereof |
| WO2022082892A1 (en) * | 2020-10-20 | 2022-04-28 | 苏州莱锦机电自动化有限公司 | Big data analysis method and system, and computer device and storage medium thereof |
| CN112527848A (en) * | 2020-12-22 | 2021-03-19 | 苏州科达科技股份有限公司 | Multi-data-source-based report data query method, device, system and storage medium |
| CN113836216A (en) * | 2021-09-06 | 2021-12-24 | 飞算数智科技(深圳)有限公司 | Data display method and device, storage medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111046083A (en) | Data analysis method and system and big data platform | |
| CN106446228B (en) | Method and device for collecting and analyzing WEB page data | |
| CN106384273B (en) | Malicious bill-swiping detection system and method | |
| CN102946319B (en) | Networks congestion control information analysis system and analytical method thereof | |
| CN105590055B (en) | Method and device for identifying user credible behaviors in network interaction system | |
| CN111681091B (en) | Financial risk prediction method and device based on time domain information and storage medium | |
| CN106980573B (en) | Method, device and system for constructing test case request object | |
| CN107563757B (en) | Data risk identification method and device | |
| CN101222348B (en) | Method and system for calculating number of website real user | |
| CN102592067B (en) | Webpage recognition method, device and system | |
| US20180157851A1 (en) | Systems and methods for authentication of access based on multi-data source information | |
| CN104836781A (en) | Method distinguishing identities of access users, and device | |
| CN104135365A (en) | A method, a server, and a client for verifying an access request | |
| CN104598557A (en) | Method and device for data rasterization and method and device for user behavior analysis | |
| CN110298662B (en) | Automatic detection method and device for transaction repeated submission | |
| KR102242782B1 (en) | Method and device for sharing local information | |
| CN110738784B (en) | Invoice management method, device, server and system | |
| CN110728301A (en) | Credit scoring method, device, terminal and storage medium for individual user | |
| CN106682206A (en) | Method and system for big data processing | |
| CN106921711A (en) | The method for pushing of automobile information, device and server | |
| CN110929153A (en) | Personalized start page pushing method, device and system | |
| CN112800127B (en) | Data mining analysis method and device based on transaction bill | |
| CN112365248B (en) | Method and system for analyzing digital currency transaction path | |
| CN112488562B (en) | Service realization method and device | |
| CN110990810B (en) | User operation data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200421 |