CN111291246A - Big data rapid analysis system - Google Patents
Big data rapid analysis system Download PDFInfo
- Publication number
- CN111291246A CN111291246A CN202010126796.0A CN202010126796A CN111291246A CN 111291246 A CN111291246 A CN 111291246A CN 202010126796 A CN202010126796 A CN 202010126796A CN 111291246 A CN111291246 A CN 111291246A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- cache server
- central processing
- calling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/906—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9038—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a big data rapid analysis system, and belongs to the field of big data. The technical scheme of the invention is as follows: the data collection module can collect all behavior data of a user on the user terminal in a unified manner, then the data collected in the data collection module in the unified manner are classified according to different indexes through the data classification module, the data classification is realized firstly, the data subjected to preliminary classification are directly stored and called, the behavior of primarily selecting and calling required data from all disordered data is effectively saved, the data calling time is shortened, before the data are transmitted to the central processing through the data output module, the classified data called through the data calling module are compared with the original data stored in the data cache server through the data comparison module, the data damage is avoided, the lossy data are prevented from being called, and the analysis result error caused by the error of the data calling is avoided.
Description
Technical Field
The invention belongs to the field of big data, and particularly relates to a big data rapid analysis system.
Background
Big data, an IT industry term, refers to a data set that cannot be captured, managed, and processed with a conventional software tool within a certain time range, and is a massive, high-growth-rate, and diversified information asset that needs a new processing mode to have stronger decision-making power, insight discovery power, and process optimization capability. With the advent of the cloud era, big data has attracted more and more attention. Analyst teams believe that large data is often used to describe the large amount of unstructured and semi-structured data created by a company that can take excessive time and money to download to a relational database for analysis. Big data analysis is often tied to cloud computing because real-time large dataset analysis requires a MapReduce-like framework to distribute work to tens, hundreds, or even thousands of computers. However, the current data analysis is based on the fact that all data are analyzed together in a general manner, and time is consumed. Based on the scheme of the invention, a big data rapid analysis system needs to be designed to solve the problems.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the invention aims to provide a big data rapid analysis system, so as to solve the problem that the existing data analysis takes long time because the analysis is carried out on the basis of all data together.
In order to solve the technical problems, the invention adopts the following technical scheme:
a big data rapid analysis system; the system comprises a data collection module, a data classification module, a data cache server, a central processing unit and a user terminal; the signal output end of the data collection module is respectively connected with the signal input ends of the data classification module and the data cache server, the signal output end of the data classification module is connected with the signal input end of the data cache server, the signal output end of the data cache server is connected with the signal input end of the central processing unit, and the signal output end of the central processing unit is connected with the signal input end of the user terminal.
The data classification module divides data into a plurality of groups according to different categories and transmits the groups to the data cache server, wherein different data groups have a data superposition phenomenon.
Wherein, the data cache server comprises original data, classified data, a data calling module, a data comparison module and a data output module, the original data is collected by the data collection module, the collected data is stored in the data cache server to generate the original data, the classified data does not classify the data collected by the data collection module by the data classification module, the classified data is stored in the data cache server again according to different categories to generate the classified data, the signal input end of the data calling module is connected with the signal output end of the classified data, the signal input end of the data comparison module is respectively connected with the data calling module and the original data, the signal output end of the data comparison module is connected with the signal input end of the data output module, the data calling module calls the classified data, the data comparison module compares the original data with the classification data called by the data calling signal, and the two groups of data are output to the central processing unit through the data output module after comparison is completed.
The central processing unit comprises a data processing module and a data processing record generating module, wherein the signal output end of the data processing module is respectively connected with the data processing record generating module and the signal input end of the user terminal, the data processing module receives the data output by the data output module for processing, and the data processing record generating module processes, records and stores the data output by the data processing module to the data output by the output module.
The user terminal comprises a visualization module, a user control module and a data modification module, the user control module calls data processed by the central processing unit, the data is displayed in an image form through the visualization module, the data modification module modifies the data processed by the central processing unit on line and stores the modified data in the user terminal to generate a modification record.
The user terminal receives data output by the central processing unit through the network transmission module.
Compared with other methods, the method has the beneficial technical effects that:
in the system provided by the scheme of the invention, all behavior data of a user on a user terminal can be uniformly collected through the data collection module, then the data uniformly collected in the data collection module is classified according to different indexes through the data classification module, the data classification is realized firstly, the data subjected to the primary classification is directly stored for calling, the behavior of primarily selecting and calling required data from all disordered data is effectively saved, the data calling time is shortened, before the data is transmitted to a central processing unit through the data output module, the classified data called through the data calling module is compared with the original data stored in the data cache server through the data comparison module, the data damage is avoided, the lossy data is prevented from being called, and the analysis result error caused by the error of the data calling is avoided.
Drawings
FIG. 1 is a system configuration diagram of the present invention;
FIG. 2 is a diagram of a data caching server according to the present invention;
FIG. 3 is a structural diagram of a CPU according to the present invention;
fig. 4 is a configuration diagram of a user terminal in the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The big data rapid analysis system provided by the invention comprises a data collection module, a data classification module, a data cache server, a central processing unit and a user terminal, wherein a signal output end of the data collection module is respectively connected with signal input ends of the data classification module and the data cache server, a signal output end of the data classification module is connected with a signal input end of the data cache server, a signal output end of the data cache server is connected with a signal input end of the central processing unit, and a signal output end of the central processing unit is connected with a signal input end of the user terminal.
The data classification module divides data into a plurality of groups according to different categories and transmits the groups to the data cache server, and different data groups have a data superposition phenomenon.
The data cache server comprises original data, classified data, a data calling module, a data comparison module and a data output module, wherein the original data is obtained by completing data collection through the data collection module, the collected data is stored in the data cache server to generate the original data, the classified data does not classify the data collected by the data collection module through the data classification module, the classified data is stored in the data cache server again according to different categories to generate the classified data, the signal input end of the data calling module is electrically connected with the signal output end of the classified data, the signal input end of the data comparison module is respectively electrically connected with the data calling module and the original data, and the signal output end of the data comparison module is electrically connected with the signal input end of the data output module, the data calling module calls the classification data, the data comparison module compares the original data with the classification data called by the data calling signal, and the two groups of data are output to a central processing unit through the data output module after comparison is completed.
The central processing unit comprises a data processing module and a data processing record generating module, wherein the signal output end of the data processing module is electrically connected with the data processing record generating module and the signal input end of the user terminal respectively, the data processing module receives the data output by the data output module for processing, and the data processing record generating module processes, generates, records and stores the data output by the data processing module.
The user terminal comprises a visualization module, a user control module and a data modification module, wherein the user control module calls data processed by the central processing unit, the data is displayed through the visualization module, the data modification module modifies the data processed by the central processing unit on line and stores the modified data in the user terminal to generate a modification record.
And the user terminal receives the data output by the central processing unit through a network transmission module.
The workflow of the scheme of the embodiment is as follows:
the data collection module collects all behavior data of a user on a user terminal in a unified manner, the uniformly collected data are stored in the data cache server as original data, the uniformly collected data in the data collection module are classified according to different indexes through the data classification module, the classified data can be overlapped, the classified data are stored in the data cache server as classified data again, the classification of the data is realized firstly, the primarily classified data are stored and called directly, the behavior of primarily selecting and calling required data from all disordered data is effectively saved, the data calling time is effectively saved, the classified data are called according to requirements through the data calling module, before the data are transmitted to the central processing through the data output module, the classified data called through the data calling module and the original data stored in the data cache server are paired through the data comparison module Compared with the prior art, the method has the advantages that data damage is avoided, destructive data are prevented from being called, analysis result errors caused by data calling errors are avoided, the central processing unit is used for processing data, the processing record is generated through the data processing record generating module, a user calls the analysis data at a user terminal, the data analysis result is finally provided for the user to use, and the data can be modified through the data modifying module.
The present invention has been further described with reference to specific embodiments, but it should be understood that the detailed description should not be construed as limiting the spirit and scope of the present invention, and various modifications made to the above-described embodiments by those of ordinary skill in the art after reading this specification are within the scope of the present invention.
Claims (6)
1. A big data rapid analysis system is characterized by comprising a data collection module, a data classification module, a data cache server, a central processing unit and a user terminal; the signal output end of the data collection module is respectively connected with the signal input ends of the data classification module and the data cache server, the signal output end of the data classification module is connected with the signal input end of the data cache server, the signal output end of the data cache server is connected with the signal input end of the central processing unit, and the signal output end of the central processing unit is connected with the signal input end of the user terminal.
2. The big data rapid analysis system according to claim 1, wherein the data classification module divides the data into a plurality of groups according to different categories and transmits the groups to the data cache server, wherein the data superposition phenomenon exists in different data groups.
3. The big data rapid analysis system according to claim 2, wherein the data cache server comprises original data, classified data, a data calling module, a data comparing module and a data outputting module, the original data is obtained by collecting data through the data collecting module and storing the collected data in the data cache server to generate original data, the classified data does not classify the data collected by the data collecting module through the data classifying module, the classified data is stored in the data cache server again according to different categories to generate classified data, the signal input end of the data calling module is connected with the signal output end of the classified data, the signal input end of the data comparing module is respectively connected with the data calling module and the original data, the signal output end of the data comparing module is connected with the signal input end of the data outputting module, the data calling module calls the classification data, the data comparison module compares the original data with the classification data called by the data calling signal, and the two groups of data are output to the central processing unit through the data output module after comparison is completed.
4. The big data rapid analysis system according to claim 3, wherein the central processing unit comprises a data processing module and a data processing record generation module, a signal output end of the data processing module is respectively connected with the data processing record generation module and a signal input end of the user terminal, the data processing module receives and processes data output by the data output module, and the data processing record generation module processes, records and stores the data output by the output module.
5. The big data rapid analysis system according to claim 4, wherein the user terminal comprises a visualization module, a user control module and a data modification module, the user control module calls the data processed by the central processing unit, the data is displayed in an image form through the visualization module, the data modification module modifies the data processed by the central processing unit on line and stores the modified data in the user terminal to generate a modification record.
6. The big data rapid analysis system according to claim 5, wherein the user terminal receives data output by the central processing unit through the network transmission module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010126796.0A CN111291246A (en) | 2020-02-28 | 2020-02-28 | Big data rapid analysis system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010126796.0A CN111291246A (en) | 2020-02-28 | 2020-02-28 | Big data rapid analysis system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111291246A true CN111291246A (en) | 2020-06-16 |
Family
ID=71028337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010126796.0A Withdrawn CN111291246A (en) | 2020-02-28 | 2020-02-28 | Big data rapid analysis system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111291246A (en) |
-
2020
- 2020-02-28 CN CN202010126796.0A patent/CN111291246A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111240662B (en) | Spark machine learning system and method based on task visual drag | |
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| EP3788560A1 (en) | Systems and methods for enriching modeling tools and infrastructure with semantics | |
| CN111984499A (en) | Fault detection method and device for big data cluster | |
| CN111339073A (en) | Real-time data processing method and device, electronic equipment and readable storage medium | |
| CN108415964A (en) | Tables of data querying method, device, terminal device and storage medium | |
| US20190050435A1 (en) | Object data association index system and methods for the construction and applications thereof | |
| CN112948492A (en) | Data processing system, method and device, electronic equipment and storage medium | |
| CN114461644A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN114880405A (en) | Data lake-based data processing method and system | |
| CN112817958A (en) | Electric power planning data acquisition method and device and intelligent terminal | |
| CN113590607A (en) | Electric power marketing report realization method and system based on report factor | |
| CN113570468A (en) | Enterprise payment wind control service platform | |
| CN104199889A (en) | RTLogic big data processing system and method based on CEP technology | |
| CN116701525A (en) | Early warning method and system based on real-time data analysis and electronic equipment | |
| CN116719799A (en) | Environment-friendly data management method, device, computer equipment and storage medium | |
| CN111291246A (en) | Big data rapid analysis system | |
| CN101546395A (en) | Method and equipment for generating service process model and method for identifying status field | |
| CN112825165A (en) | Project quality management method and device | |
| CN111130882A (en) | Monitoring system and method of network equipment | |
| CN114281494A (en) | Data full life cycle management method, system, terminal device and storage medium | |
| CN114022295A (en) | Group fraud identification method and system | |
| CN114625763A (en) | Information analysis method and device for database, electronic equipment and readable medium | |
| CN111581254A (en) | ETL method and system based on internet financial data | |
| CN112035366A (en) | Test case generation method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20200616 |
|
| WW01 | Invention patent application withdrawn after publication |