CN116501779A - Big data mining analysis system for real-time feedback - Google Patents
Big data mining analysis system for real-time feedback Download PDFInfo
- Publication number
- CN116501779A CN116501779A CN202310755665.2A CN202310755665A CN116501779A CN 116501779 A CN116501779 A CN 116501779A CN 202310755665 A CN202310755665 A CN 202310755665A CN 116501779 A CN116501779 A CN 116501779A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- real
- market
- report
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 71
- 238000007418 data mining Methods 0.000 title claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 21
- 238000013480 data collection Methods 0.000 claims abstract description 15
- 230000005540 biological transmission Effects 0.000 claims abstract description 11
- 238000010586 diagram Methods 0.000 claims description 18
- 230000008451 emotion Effects 0.000 claims description 18
- 238000004140 cleaning Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 13
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 239000013598 vector Substances 0.000 claims description 8
- 230000004913 activation Effects 0.000 claims description 6
- 238000011156 evaluation Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 230000000007 visual effect Effects 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 3
- 238000007405 data analysis Methods 0.000 abstract description 4
- 238000005065 mining Methods 0.000 abstract description 3
- 230000006872 improvement Effects 0.000 description 7
- 238000000034 method Methods 0.000 description 5
- 230000015654 memory Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011157 data evaluation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the technical field of big data analysis. The invention relates to a big data mining analysis system for real-time feedback. The system comprises a data collection unit, a distributed storage unit, a real-time processing unit, a data transmission unit and a scheme updating unit; the data collection unit is used for collecting market data and processing and analyzing the collected data; according to the invention, the collected data is rapidly classified and analyzed in real time, so that the arrival of the data can be responded immediately, the mining and analysis results are generated in real time, the user can acquire the latest data analysis results in time, decisions and actions can be better supported, the accuracy and instantaneity of the analysis results are ensured by evaluating the cleaned data, the analysis parameters and strategies are flexibly set by the user by collecting the user feedback demand data, the analysis results are intuitively displayed, and the user can rapidly understand and apply the analysis results.
Description
Technical Field
The invention relates to the technical field of big data mining analysis, in particular to a big data mining analysis system for real-time feedback.
Background
With the advent of the big data age, a large amount of data is generated and accumulated, and by using the data to perform effective mining and analysis, the association and rule hidden behind the data can be revealed, and powerful support is provided for decision making, however, in terms of real-time performance and feedback performance, the traditional big data mining analysis system has slower real-time analysis display of the data due to the repetition of the data and the difference of the classification of the data, and in view of this, a big data mining analysis system for performing real-time feedback is proposed.
Disclosure of Invention
The invention aims to provide a big data mining analysis system for real-time feedback so as to solve the problems in the background technology.
In order to achieve the above purpose, a big data mining analysis system for real-time feedback is provided, which comprises a data collection unit, a distributed storage unit, a real-time processing unit, a data transmission unit and a scheme updating unit;
the data collection unit is used for collecting market data, processing and analyzing the collected data, and the distribution storage unit is used for classifying and marking the data collected by the data collection unit and distributing the data to each operation node;
the real-time processing unit is used for analyzing each operation node of the distributed storage unit and generating a market trend report according to analysis results and evaluation;
the data transmission unit is used for performing display conversion on the market trend report generated by the real-time processing unit and uploading the market trend report to the cloud;
the scheme updating unit is used for collecting user cloud feedback data and updating by combining with the operation node.
As a further improvement of the technical scheme, the data collection unit comprises an information acquisition module and a data cleaning module;
the information acquisition module is used for acquiring market data by connecting an API port to a market supplier;
the data cleaning module is used for cleaning according to market data acquired by the information acquisition module.
As a further improvement of the technical scheme, the distributed storage unit comprises a data classification module and a data distribution module;
the data classification module is used for evaluating the data cleaned by the data cleaning module and classifying the data according to the evaluation result;
the data distribution module is used for carrying out distributed storage on market data by adopting a distributed storage system according to the classification result of the data classification module.
As a further improvement of the technical scheme, the expression of the data distribution module for carrying out distributed storage on market data by adopting a distributed storage system is as follows:
；
the smallest of them->；
Wherein,,: mapping a certain node i into a probability space of 0-1;
: mapping data d to a probability space;
: the number of nodes is stored in a distributed manner, and the data d is stored in the node n.
The real-time processing unit comprises a characteristic acquisition module and a trend prediction module;
the characteristic acquisition module is used for carrying out characteristic extraction and emotion analysis on the data stored in the distribution;
and the trend prediction module performs trend prediction analysis by combining the analysis data of the characteristic acquisition module with the market data, and generates a market trend report according to the analysis data.
As a further improvement of the technical scheme, the step of generating the market trend report by the trend prediction module according to the analysis data is as follows:
on the results of feature extraction and emotion analysis, trend prediction is performed using an RNN model, in which time series data can be input into each time series step of the RNN so that the network makes predictions of the next time step according to the previous information of the sequence, the formula is as follows:
；
wherein,,is a market trend predictive hidden state vector representing the time step of network learning +.>Information of->Is the input sequence at time step->Value of->And->Is a weight matrix representing the weights of the hidden state and the input state, respectively, < >>Is a deviation term->Representing an activation function->Is RNN at time step->Output of->Is an output weight matrix,/>Is the output deviation of the output of the device,is an activation function for classifying tasks, converting the output into a probability value.
As a further improvement of the technical scheme, the data transmission unit comprises a report generation module and a data encryption module;
the report generation module is used for carrying out visual display format conversion on the market trend report generated by the trend prediction module;
the data encryption module is used for uploading the converted report to the cloud end in a digital encryption mode.
As a further improvement of the technical scheme, the step of carrying out visual display format conversion on the market trend report by the report generation module is as follows:
visually displaying the data in a column diagram, a line diagram, a scatter diagram and a cake diagram;
line graph: for displaying successive data changes over a period of time, such as stock prices, sales;
figure: for comparing data differences between different categories, such as sales of different products;
scatter plot: for displaying the correlation between two variables, such as the correlation between revenue and cost;
pie chart: for displaying the proportion of different categories of data to the total, such as market share.
As a further improvement of the technical scheme, the scheme updating unit comprises a requirement acquisition module and a report updating module;
the demand acquisition module is used for acquiring data fed back by a user after the user views the report, and evaluating the data to generate analysis demand data;
the report updating module is used for transmitting the analysis demand data generated by the demand acquisition module to the market trend report generated by the characteristic acquisition module for data replacement and updating, and acquiring a new market trend report.
Compared with the prior art, the invention has the beneficial effects that:
in the real-time feedback big data mining analysis system, collected data is rapidly classified and analyzed in real time, so that the arrival of the data can be responded immediately, mining and analysis results are generated in real time, a user can acquire the latest data analysis results in time, decisions and actions can be better supported, the accuracy and instantaneity of the analysis results are guaranteed by evaluating the cleaned data, and the user can flexibly set analysis parameters and strategies by collecting user feedback demand data so as to meet different analysis demands, intuitively display the analysis results, and rapidly understand and apply the analysis results.
Drawings
FIG. 1 is a schematic diagram of the overall structure of the present invention;
FIG. 2 is a schematic diagram of the structure of the present invention for collecting market data;
FIG. 3 is a schematic diagram of the architecture of the present invention assigned to each operational node;
FIG. 4 is a structural schematic diagram of the present invention for generating market trend reports;
FIG. 5 is a schematic diagram of a display conversion structure according to the present invention;
fig. 6 is a schematic diagram of the present invention for updating in conjunction with an operational node.
The meaning of each reference sign in the figure is:
10. a data collection unit; 11. an information acquisition module; 12. a data cleaning module;
20. a distribution storage unit; 21. a data classification module; 22. a data distribution module;
30. a real-time processing unit; 31. the characteristic acquisition module; 32. a trend prediction module;
40. a data transmission unit; 41. a report generation module; 42. a data encryption module;
50. a scheme updating unit; 51. a demand acquisition module; 52. and a report updating module.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Examples: referring to fig. 1-6, the present embodiment is directed to a big data mining analysis system for real-time feedback, which includes a data collection unit 10, a distribution storage unit 20, a real-time processing unit 30, a data transmission unit 40 and a scheme updating unit 50;
the data collection unit 10 is used for collecting market data and processing and analyzing the collected data;
the data collection unit 10 includes an information collection module 11 and a data cleaning module 12;
the information acquisition module 11 is used for acquiring market data by using an API port connection for a market supplier; collecting real-time market data streams, including transaction data, social media data, and news data, through an API interface connection with a market data provider;
the data cleaning module 12 is used for cleaning according to market data acquired by the information acquisition module 11. The cleaning steps are as follows:
transaction data cleaning:
deleting incomplete or invalid transaction records;
removing the duplicate transaction records;
formatting transaction times for analysis;
processing abnormal transaction data, such as transaction prices or volume outside of normal range;
the transaction data is converted into a format suitable for analysis, such as a time series format.
Social media data cleansing:
deleting irrelevant or advertisement information;
extracting word stems and marking parts of speech of the text;
handling unusual social media data, such as comments violating regulations, and spurious information, etc.;
social media data is converted into a format suitable for analysis, such as text data into Word2Vec vectors.
News data cleaning:
deleting useless, duplicate or non-canonical data such as news stories in a wrong format;
keyword extraction, classification, emotion analysis and other treatments are carried out on news content;
processing unusual news data such as an erroneous time stamp, news stories that have been revised, and the like;
the news data is converted into a format suitable for analysis, for example text data is converted into LDA topic distribution vectors.
The distribution storage unit 20 is used for classifying and marking according to the data collected by the data collection unit 10 and distributing the data to each operation node; the distribution storage unit 20 includes a data classification module 21 and a data distribution module 22;
the data classification module 21 is used for evaluating the data cleaned by the data cleaning module 12 and classifying the data according to the evaluation result; checking duplicate according to the cleaned data, avoiding useless repeated data, and enabling the data to be according to transaction data of a plurality of small memories, social media data of the small memories and news data of the small memories;
the data distribution module 22 is configured to distribute and store market data by using a distributed storage system according to the classification result of the data classification module 21.
Selecting a distributed storage system:
and selecting a proper distributed storage system according to the strategy of data distributed storage. For transactional data, social media data, and news data, different distributed storage systems are typically employed, as follows:
transaction data: a distributed database based on column storage, HBase, cassandra, etc., may be employed;
social media data: a distributed document database MongoDB, couchDB and the like can be adopted to carry out full text indexing in combination with a search engine ElasticSearch, solr and the like;
news data: a distributed database, such as Apache, may be employed that is specific to text storage and analysis.
The expression of the data distribution module 22 for distributing and storing market data by using the distributed storage system is as follows:
；
the smallest of them->；
Wherein,,: mapping a certain node i into a probability space of 0-1;
: mapping data d to a probability space;
: the number of nodes is stored in a distributed manner, and the data d is stored in the node n.
The real-time processing unit 30 is used for analyzing the inside of each operation node of the distribution storage unit 20 and generating a market trend report according to the analysis result and the evaluation;
the real-time processing unit 30 comprises a feature acquisition module 31 and a trend prediction module 32;
the feature collection module 31 is used for carrying out feature extraction and emotion analysis on the data stored in the distribution;
feature extraction:
and extracting the characteristics of the data, and converting the data into representative characteristic vectors for subsequent emotion analysis and machine learning algorithm processing. The feature extraction method comprises text feature extraction, image feature extraction, audio feature extraction and the like. Taking text features as an example, the following method may be adopted:
word bag model (BagofWords): text information is represented as a vector containing vocabulary items, one dimension for each vocabulary item. The text information is then represented as this vector. This can be achieved using the scikit-learn library of Python.
Tf-Idf (terminal-inverse document frequency): on the basis of the word bag model, the importance degree of each vocabulary item is considered, and the higher the word frequency is, the lower the importance is. This can be achieved using the scikit-learn library of Python.
Emotion analysis: and carrying out emotion analysis on the data after the feature extraction, namely analyzing emotion polarity and emotion strength in the data. Emotion analysis may employ supervised and unsupervised machine learning algorithms and rule-based methods. Taking text emotion analysis as an example, the following method may be adopted: -based on an emotion dictionary: and calculating the number and weight of the positive words and the negative words in the text by applying the emotion dictionary to the text, so as to obtain the emotion score of the text. The trend prediction module 32 performs trend prediction analysis on the analysis data of the feature collection module 31 in combination with market data by using a Chinese word bank and an emotion dictionary such as HanLP and Jieba, and generates a market trend report according to the analysis data.
The trend prediction module 32 generates a market trend report from the analysis data as follows:
on the results of feature extraction and emotion analysis, trend prediction is performed using an RNN model, in which time series data can be input into each time series step of the RNN so that the network makes predictions of the next time step according to the previous information of the sequence, the formula is as follows:
；
wherein,,is a market trend predictive hidden state vector representing the time step of network learning +.>Information of->Is the input sequence at time step->Value of->And->Is a weight matrix representing the weights of the hidden state and the input state, respectively, < >>Is a deviation term->Representing an activation function->Is RNN at time step->Output of->Is an output weight matrix,/>Is the output deviation of the output of the device,is an activation function for classifying tasks, converting the output into a probability value.
The data transmission unit 40 is used for performing display conversion on the market trend report generated by the real-time processing unit 30 and uploading the market trend report to the cloud;
the data transmission unit 40 includes a report generation module 41 and a data encryption module 42;
the report generating module 41 is configured to perform visual display format conversion on the market trend report generated by the trend predicting module 32;
the steps of the report generating module 41 for visual display format conversion of market trend reports are as follows:
visually displaying the data in a column diagram, a line diagram, a scatter diagram and a cake diagram;
line graph: for displaying successive data changes over a period of time, such as stock prices, sales;
figure: for comparing data differences between different categories, such as sales of different products;
scatter plot: for displaying the correlation between two variables, such as the correlation between revenue and cost;
pie chart: for displaying the proportion of different categories of data to the total, such as market share.
The data encryption module 42 is configured to upload the converted report to the cloud end in a digital encryption manner.
The scheme updating unit 50 is configured to collect user cloud feedback data, and update the user cloud feedback data in combination with an operation node.
The scenario updating unit 50 includes a requirement acquisition module 51 and a report updating module 52;
the demand acquisition module 51 is used for acquiring data fed back by a user after the user views the report, and evaluating the data to generate analysis demand data;
designing a feedback form: first, a feedback form needs to be designed, and feedback information provided by a user after viewing a report is collected. Forms may include a number of questions, such as: "how do you overall satisfaction with the content of the report? "do you clearly understand the conclusion of the report? "etc., the user can choose the corresponding option or fill in the suggestion of his opinion according to his opinion.
Setting feedback data storage: after the feedback information of the user is collected, it needs to be stored for subsequent analysis. The storage may be performed using techniques such as databases.
Data analysis and evaluation: the collected feedback data is analyzed and evaluated. The data visualization technology, such as drawing a histogram, a pie chart and the like, can be used for clearly showing answer results of different questions, various feedback duty ratios, trends and the like, and acquiring the direction of the user needing specific refinement data;
the report updating module 52 is configured to send the analysis demand data generated by the demand acquisition module 51 to the market trend report generated by the feature acquisition module 31 for data replacement and updating, so as to obtain a new market trend report. The method comprises the following steps:
determining a data location for replacement update: it is first necessary to determine the data locations in the market trend report that require replacement updates, and to mark them in the market trend report. The data placeholders that need to be replaced and updated may be contained using a particular tag or format, for example using tags such as { }' or < > >.
Obtaining market trend report data: the data of the market trend report needs to be acquired, and the request for the market trend report can be made by using the requests library of Python, and the content of the market trend report is acquired.
Realizing replacement and update of data: and embedding new data obtained by analyzing the demand into a market trend report to finish the replacement and update of the data.
Saving new market trend reports: storing the updated market trend report to a cloud for storage for a user to check;
the foregoing has shown and described the basic principles, principal features and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the above-described embodiments, and that the above-described embodiments and descriptions are only preferred embodiments of the present invention, and are not intended to limit the invention, and that various changes and modifications may be made therein without departing from the spirit and scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (9)
1. A big data mining analysis system for real-time feedback is characterized in that: comprises a data collection unit (10), a distributed storage unit (20), a real-time processing unit (30), a data transmission unit (40) and a scheme updating unit (50);
the data collection unit (10) is used for collecting market data, processing and analyzing the collected data, and the distribution storage unit (20) is used for classifying and marking according to the data collected by the data collection unit (10) and distributing the data to each operation node;
the real-time processing unit (30) is used for analyzing the inside of each operation node of the distributed storage unit (20) and generating a market trend report according to the analysis result and the evaluation;
the data transmission unit (40) is used for performing display conversion on the market trend report generated by the real-time processing unit (30) and uploading the market trend report to the cloud;
the scheme updating unit (50) is used for collecting user cloud feedback data and updating by combining with an operation node.
2. The real-time feedback big data mining analysis system of claim 1, wherein: the data collection unit (10) comprises an information acquisition module (11) and a data cleaning module (12);
the information acquisition module (11) is used for acquiring market data for market suppliers by using API port connection;
the data cleaning module (12) is used for cleaning according to market data acquired by the information acquisition module (11).
3. The real-time feedback big data mining analysis system of claim 2, wherein: the distribution storage unit (20) comprises a data classification module (21) and a data distribution module (22);
the data classification module (21) is used for evaluating the data cleaned by the data cleaning module (12) and classifying the data according to the evaluation result;
the data distribution module (22) is used for distributing and storing market data by adopting a distributed storage system according to the classification result of the data classification module (21).
4. A real time feedback big data mining analysis system according to claim 3, wherein: the data distribution module (22) adopts a distributed storage system to carry out distributed storage on market data according to the following expression:
；
the smallest of them->；
Wherein,,: mapping a certain node i into a probability space of 0-1;
: mapping data d to a probability space;
: the number of nodes is stored in a distributed manner, and the data d is stored in the node n.
5. The real-time feedback big data mining analysis system of claim 1, wherein: the real-time processing unit (30) comprises a characteristic acquisition module (31) and a trend prediction module (32);
the characteristic acquisition module (31) is used for carrying out characteristic extraction and emotion analysis on the data stored in the distribution;
the trend prediction module (32) performs trend prediction analysis on the analysis data of the characteristic acquisition module (31) in combination with market data, and generates a market trend report according to the analysis data.
6. The real-time feedback big data mining analysis system of claim 5, wherein: the trend prediction module (32) generates a market trend report from the analysis data as follows:
on the results of feature extraction and emotion analysis, trend prediction is performed using an RNN model, in which time series data can be input into each time series step of the RNN so that the network makes predictions of the next time step according to the previous information of the sequence, the formula is as follows:
；
wherein,,is a market trend predictive hidden state vector representing the time step of network learning +.>Information of->Is the input sequence at time step->Value of->And->Is a weight matrix representing the weights of the hidden state and the input state, respectively, < >>Is a deviation term->Representing an activation function->Is RNN at time step->Output of->Is an output weight matrix,/>Is the output deviation of the output of the device,is an activation function for classifying tasks, converting the output into a probability value.
7. The real-time feedback big data mining analysis system of claim 5, wherein: the data transmission unit (40) comprises a report generation module (41) and a data encryption module (42);
the report generation module (41) is used for performing visual display format conversion on the market trend report generated by the trend prediction module (32);
the data encryption module (42) is used for uploading the converted report to the cloud end in a digital encryption mode.
8. The real-time feedback big data mining analysis system of claim 7, wherein: the visual display format conversion step of the market trend report of the report generation module (41) is as follows:
visually displaying the data in a column diagram, a line diagram, a scatter diagram and a cake diagram;
line graph: for displaying successive data changes over a period of time, such as stock prices, sales;
figure: for comparing data differences between different categories, such as sales of different products;
scatter plot: for displaying the correlation between two variables, such as the correlation between revenue and cost;
pie chart: for displaying the proportion of different categories of data to the total, such as market share.
9. The real-time feedback big data mining analysis system of claim 1, wherein: the scheme updating unit (50) comprises a requirement acquisition module (51) and a report updating module (52);
the demand acquisition module (51) is used for acquiring data fed back by a user after the user views the report, and evaluating the data to generate analysis demand data;
the report updating module (52) is used for transmitting the analysis demand data generated by the demand acquisition module (51) to the market trend report generated by the characteristic acquisition module (31) for data replacement updating, and acquiring a new market trend report.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310755665.2A CN116501779A (en) | 2023-06-26 | 2023-06-26 | Big data mining analysis system for real-time feedback |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310755665.2A CN116501779A (en) | 2023-06-26 | 2023-06-26 | Big data mining analysis system for real-time feedback |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116501779A true CN116501779A (en) | 2023-07-28 |
Family
ID=87320510
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310755665.2A Pending CN116501779A (en) | 2023-06-26 | 2023-06-26 | Big data mining analysis system for real-time feedback |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116501779A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117196694A (en) * | 2023-11-02 | 2023-12-08 | 北京法伯宏业科技发展有限公司 | Medicine market data analysis method and system based on big data |
| CN117591578A (en) * | 2024-01-18 | 2024-02-23 | 山东科技大学 | Data mining system and mining method based on big data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107919983A (en) * | 2017-11-01 | 2018-04-17 | 中国科学院软件研究所 | A kind of space information network Effectiveness Evaluation System and method based on data mining |
| CN108364124A (en) * | 2018-01-26 | 2018-08-03 | 天津中科智能识别产业技术研究院有限公司 | International production capacity Cooperation Risk assessment based on big data and Decision Making Service System |
| CN109033497A (en) * | 2018-06-04 | 2018-12-18 | 南瑞集团有限公司 | A kind of multistage data mining algorithm intelligent selecting method towards high concurrent |
| CN109581981A (en) * | 2018-12-06 | 2019-04-05 | 山东大学 | A kind of data fusion system and its working method based on data assessment Yu system coordination module |
| CN111723277A (en) * | 2019-03-19 | 2020-09-29 | 北京赛迈特锐医疗科技有限公司 | Image structured report cloud requirement classification system and method thereof |
| CN113177698A (en) * | 2021-04-12 | 2021-07-27 | 北京科技大学 | Industrial big data analysis aid decision platform system |
-
2023
- 2023-06-26 CN CN202310755665.2A patent/CN116501779A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107919983A (en) * | 2017-11-01 | 2018-04-17 | 中国科学院软件研究所 | A kind of space information network Effectiveness Evaluation System and method based on data mining |
| CN108364124A (en) * | 2018-01-26 | 2018-08-03 | 天津中科智能识别产业技术研究院有限公司 | International production capacity Cooperation Risk assessment based on big data and Decision Making Service System |
| CN109033497A (en) * | 2018-06-04 | 2018-12-18 | 南瑞集团有限公司 | A kind of multistage data mining algorithm intelligent selecting method towards high concurrent |
| CN109581981A (en) * | 2018-12-06 | 2019-04-05 | 山东大学 | A kind of data fusion system and its working method based on data assessment Yu system coordination module |
| CN111723277A (en) * | 2019-03-19 | 2020-09-29 | 北京赛迈特锐医疗科技有限公司 | Image structured report cloud requirement classification system and method thereof |
| CN113177698A (en) * | 2021-04-12 | 2021-07-27 | 北京科技大学 | Industrial big data analysis aid decision platform system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117196694A (en) * | 2023-11-02 | 2023-12-08 | 北京法伯宏业科技发展有限公司 | Medicine market data analysis method and system based on big data |
| CN117591578A (en) * | 2024-01-18 | 2024-02-23 | 山东科技大学 | Data mining system and mining method based on big data |
| CN117591578B (en) * | 2024-01-18 | 2024-04-09 | 山东科技大学 | Data mining system and mining method based on big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10929348B2 (en) | Method and system for large scale data curation | |
| CN111191092B (en) | Label determining method and label determining model training method | |
| CN110232109A (en) | A kind of Internet public opinion analysis method and system | |
| CN116501779A (en) | Big data mining analysis system for real-time feedback | |
| CN111489800A (en) | Analysis method and system for identifying and storing medical record and report list images | |
| US20170243125A1 (en) | Bayesian classification algorithm modification for sentiment estimation | |
| CN116468460B (en) | Consumer finance customer image recognition system and method based on artificial intelligence | |
| CN111414520A (en) | Intelligent mining system for sensitive information in public opinion information | |
| CN117235281B (en) | Multi-element data management method and system based on knowledge graph technology | |
| CN116542800A (en) | Intelligent financial statement analysis system based on cloud AI technology | |
| CN115063035A (en) | Customer evaluation method, system, equipment and storage medium based on neural network | |
| US20220198274A1 (en) | Method and system for unstructured information analysis using a pipeline of ml algorithms | |
| CN117764724A (en) | Intelligent credit rating report construction method and system | |
| CN113379432A (en) | Sales system customer matching method based on machine learning | |
| CN111753151A (en) | Service recommendation method based on internet user behaviors | |
| CN114445043B (en) | Open ecological cloud ERP-based heterogeneous graph user demand accurate discovery method and system | |
| CN114357242A (en) | Training evaluation method and device based on recall model, equipment and storage medium | |
| US11115440B2 (en) | Dynamic threat intelligence detection and control system | |
| CN113870998A (en) | Interrogation method, device, electronic equipment and storage medium | |
| KR20220105792A (en) | AI-based Decision Making Support System utilizing Dynamic Text Sources | |
| Nagarkar et al. | Social Media Intelligence for Brand Analysis | |
| US11907500B2 (en) | Automated processing and dynamic filtering of content for display | |
| CN117540004B (en) | Industrial domain intelligent question-answering method and system based on knowledge graph and user behavior | |
| US20240119295A1 (en) | Generalized Bags for Learning from Label Proportions | |
| AU2021106572A4 (en) | A recommendation system and method for e-commerce using machine learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230728 |
|
| RJ01 | Rejection of invention patent application after publication |