CN110781362A - Shopping guide platform big data processing system - Google Patents

Shopping guide platform big data processing system Download PDF

Info

Publication number
CN110781362A
CN110781362A CN201911033638.4A CN201911033638A CN110781362A CN 110781362 A CN110781362 A CN 110781362A CN 201911033638 A CN201911033638 A CN 201911033638A CN 110781362 A CN110781362 A CN 110781362A
Authority
CN
China
Prior art keywords
data
module
shopping guide
guide platform
big
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN201911033638.4A
Other languages
Chinese (zh)
Inventor
王鑫池
杨尚杰
何娟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wenzhou Hongxiuzhao Information Technology Co Ltd
Original Assignee
Wenzhou Hongxiuzhao Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wenzhou Hongxiuzhao Information Technology Co Ltd filed Critical Wenzhou Hongxiuzhao Information Technology Co Ltd
Priority to CN201911033638.4A priority Critical patent/CN110781362A/en
Publication of CN110781362A publication Critical patent/CN110781362A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/904—Browsing; Visualisation therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/217—Database tuning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/338—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G06F16/358—Browsing; Visualisation therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying
    • G06F16/9038—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/906—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9538—Presentation of query results

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Quality & Reliability (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention relates to the technical field of big data, in particular to a shopping guide platform big data processing system which comprises a shopping guide platform module, wherein the shopping guide platform module is in signal connection with a data acquisition module, the data acquisition module is connected with a data preprocessing module in a signal connection mode, the data preprocessing module is connected with a data analysis module, the data analysis module is connected with a data cleaning module in a signal connection mode, the data cleaning module comprises a data calculator, a data converter, a data integration module, a data transmission module and an application server, the application server is relied on, the data generated in the data analysis module is calculated by the data calculator, and the calculated data is transmitted to the data converter. The method can effectively improve the quality of the big data, is the embodiment of the process quality of the big data, and improves the consistency, the accuracy, the authenticity and the usability of the big data.

Description

Shopping guide platform big data processing system
Technical Field
The invention relates to the technical field of big data, in particular to a shopping guide platform big data processing system.
Background
Big data is a data set which cannot be captured, managed and processed by a conventional software tool within a certain time range, is a massive, high-growth rate and diversified information asset which can have stronger decision making power, insight discovery power and flow optimization capability only by a new processing mode, and has strategic significance not in mastering huge data information but in performing specialized processing on the data containing significance. In other words, if big data is compared to an industry, the key to realizing profitability in the industry is to improve the "processing ability" of the data and realize the "value-added" of the data through the "processing".
The shopping guide platform big data processing system on the existing market cannot rapidly classify, screen and calculate mass data, is not convenient for a client to check the data, and cannot be displayed to a user in a visual mode of computer graphics and images, so that the shopping guide platform big data processing system is not beneficial to popularization.
Disclosure of Invention
The invention aims to solve the defects in the prior art and provides a shopping guide platform big data processing system.
In order to achieve the purpose, the invention adopts the following technical scheme:
the design provides a big data processing system of shopping guide platform, including the shopping guide platform module, the big platform module signal connection of shopping guide has data acquisition module, and data acquisition module is connected with data preprocessing module through signal connection's mode simultaneously, data preprocessing module and data analysis module, data analysis module is connected through signal connection's mode with data cleaning module, and data cleaning module is connected with the high in the clouds database, and data cleaning module and shopping guide platform module interconnect simultaneously, data cleaning module includes data calculator, data converter, data integration module, data transmission module and application server, relies on application server, calculates the data of production in the data analysis module through data calculator, in with the data transfer to data converter after the calculation simultaneously, will not be different sources through data integration module, The data of format, characteristics and properties are rapidly concentrated through the logic function, and the concentrated data are transmitted to the cloud database through the data transmission module.
Preferably, the data acquisition module comprises an internet module, a Web database, a webpage data acquisition module and a multi-network data acquisition module, data about the shopping guide platform module in the internet module and the Web database are collected through the webpage data acquisition module, and meanwhile, the multi-network data acquisition module specifically adopts a multi-network crawler mode to collect the data, so that the timeliness quality of the collected data can be guaranteed.
Preferably, the data preprocessing module comprises a MapReduce distributed computing framework and a central processing unit, the MapReduce distributed computing framework is a batch processing distributed computing framework, massive data are analyzed and processed by the central processing unit, various structured data modules and unstructured data modules are processed at the same time, data reading and writing and moving expenses are effectively reduced, and the processing and running speed of big data is improved.
Preferably, the data analysis module comprises a data access connector module, an ETHINK data mining module, a network transmission module and a data visualization module, the data access connector module can identify the authority of the control end to data processing, and the ETHINK data mining module deeply mines big data and relational data through functions by means of the data access connector module and transmits the data to the data visualization module through the network transmission module.
Preferably, the data visualization module specifically comprises a tablet, a mobile client and a PC.
Preferably, shopping guide platform module signal connection has big data privacy and security module, and big data privacy and security module specifically include IDS control end authentication module, IDS control end authentication module signal connection has the fire wall module, the fire wall module is connected with the data entry module, and backs up in passing through the data entry module with data transfer to website data backup module.
Preferably, the shopping guide platform module is in signal connection with a big data application module and a data source collection module, the data source collection module is transmitted to the signal converter module through the wireless network transmission module, the optical signals are converted into electric signals by the signal converter module, and the electric signals are transmitted to the mobile client module, so that the client can conveniently check the data.
The shopping guide platform big data processing system provided by the invention has the beneficial effects that:
1. massive data are analyzed and processed by the central processing unit, and various structured data modules and unstructured data modules are processed at the same time, so that the data reading and writing and moving expenses are effectively reduced, and the processing and running speed of big data is improved;
2. the optical signals are converted into electric signals by the aid of the signal converter module, and the electric signals are transmitted to the mobile client module at the same time, so that the client can conveniently check data;
3. the multi-network data acquisition module specifically collects data in a multi-network crawler mode, so that the timeliness quality of the collected data can be guaranteed, and the data acquisition task can be started and stopped;
4. the method has the advantages that the method effectively reduces the expenses of data reading and writing and movement and improves the processing and operating speed of big data, meanwhile, one or more data sources are usually arranged in the data acquisition process and comprise isomorphic or heterogeneous databases, file systems and service interfaces, so that the method is easily influenced by noise data, data value loss and data collision, therefore, the collected big data set needs to be preprocessed firstly, the accuracy and the value of big data analysis and prediction results can be ensured, meanwhile, the quality of the big data can be effectively improved through the preprocessing of the big data, the method is the embodiment of the process quality of the big data, and the consistency, the accuracy, the authenticity and the usability of the big data are improved.
5. The analysis of the big data is beneficial to guiding the big data, and the usability and the value of the big data are guaranteed, so that the requirements of users are met.
Drawings
Fig. 1 is a system block diagram of a shopping guide platform big data processing system according to the present invention.
FIG. 2 is a system block diagram of a big data privacy and security module of a shopping guide platform big data processing system according to the present invention.
FIG. 3 is a system block diagram of a big data application module of a shopping guide platform big data processing system according to the present invention.
Fig. 4 is a system block diagram of a data acquisition module of a shopping guide platform big data processing system according to the present invention.
FIG. 5 is a system block diagram of a data preprocessing module of a shopping guide platform big data processing system according to the present invention.
FIG. 6 is a system block diagram of data analysis of a shopping guide platform big data processing system according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments.
Referring to fig. 1-6, a shopping guide platform big data processing system comprises a shopping guide platform module, wherein the shopping guide platform module is in signal connection with a data acquisition module, the data acquisition module is connected with a data preprocessing module in a signal connection manner, the data preprocessing module is connected with a data analysis module, the data analysis module is connected with a data cleaning module in a signal connection manner, the data cleaning module is connected with a cloud database, the data cleaning module is connected with the shopping guide platform module, the data cleaning module comprises a data calculator, a data converter, a data integration module, a data transmission module and an application server, and the application server is used for calculating data generated in the data analysis module through the data calculator, transmitting the calculated data to the data converter, and transmitting different sources through the data integration module, The data of format, characteristics and properties are rapidly concentrated through the logic function, and the concentrated data are transmitted to the cloud database through the data transmission module.
The data acquisition module comprises an internet module, a Web database, a webpage data acquisition module and a multi-network data acquisition module, data about the shopping guide platform module in the internet module and the Web database are collected through the webpage data acquisition module, meanwhile, the multi-network data acquisition module specifically adopts a multi-network crawler mode to collect the data, the timeliness quality of the collected data can be guaranteed, and starting and stopping of data acquisition tasks can be achieved.
The data preprocessing module comprises a MapReduce distributed computing frame and a central processing unit, the MapReduce distributed computing frame is a batch processing distributed computing frame, massive data is analyzed and processed by the central processing unit, various structured data modules and unstructured data modules are processed at the same time, the expenses of data reading, writing and moving are effectively reduced, the processing and operating speed of the big data is improved, one or more data sources are usually arranged in the data acquisition process, the data sources comprise isomorphic or heterogeneous databases, file systems and service interfaces, and therefore the data sources are easily influenced by noise data, data value loss and data collision, therefore, a collected big data set needs to be preprocessed firstly, the accuracy and the value of the big data analysis and prediction result can be ensured, and meanwhile, through the preprocessing of the big data, the quality of big data can be effectively improved, the quality of the big data process is reflected, and the consistency, accuracy, authenticity and usability of the big data are improved.
The data analysis module comprises a data access connector module, an ETHINK data mining module, a network transmission module and a data visualization module, the data access connector module can identify the authority of the control end to data processing, meanwhile, the ETHINK data mining module deeply mines big data and relation data through functions by means of the data access connector module, and transmits the data to the data visualization module through the network transmission module, analysis of the big data is facilitated, the guiding effect of the big data is facilitated, the usability and the value of the big data are guaranteed, and therefore the requirements of users are met.
The data visualization module specifically comprises a tablet, a mobile client and a PC (personal computer) terminal, the shopping guide platform module is in signal connection with a big data privacy and safety module, the big data privacy and safety module specifically comprises an IDS (Internet data System) control terminal authentication module, the IDS control terminal authentication module is in signal connection with a firewall module, the firewall module is connected with a data entry module, and data are transmitted to the website data backup module through the data entry module for backup, the design can effectively prevent the occurrence of data loss, the safety of the system is improved, and the protection effect is also achieved on the information and the big data of a user.
The shopping guide platform module is in signal connection with a big data application module and a data source collection module, the data source collection module is transmitted to the signal converter module through the wireless network transmission module, optical signals are converted into electric signals by means of the signal converter module, and the electric signals are transmitted to the mobile client module at the same time, so that the client can conveniently check data.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art should be considered to be within the technical scope of the present invention, and the technical solutions and the inventive concepts thereof according to the present invention should be equivalent or changed within the scope of the present invention.

Claims (7)

1. A shopping guide platform big data processing system comprises a shopping guide platform module and is characterized in that the shopping guide platform module is in signal connection with a data acquisition module, the data acquisition module is connected with a data preprocessing module in a signal connection mode, the data preprocessing module is connected with a data analysis module, the data analysis module is connected with a data cleaning module in a signal connection mode, the data cleaning module is connected with a cloud database, the data cleaning module is connected with the shopping guide platform module, the data cleaning module comprises a data calculator, a data converter, a data integration module, a data transmission module and an application server, the data generated in the data analysis module is calculated through the data calculator by means of the application server, and the calculated data is transmitted to the data converter, the data with different sources, formats, characteristics and properties are rapidly concentrated through the data integration module through the logic function, and the concentrated data are transmitted to the cloud database through the data transmission module.
2. The shopping guide platform big data processing system according to claim 1, wherein the data acquisition module comprises an internet module, a Web database, a Web page data acquisition module and a multi-network data acquisition module, the data about the shopping guide platform module in the internet module and the Web database is collected through the Web page data acquisition module, and the multi-network data acquisition module collects the data specifically in a multi-network crawler manner, so that the timeliness quality of the collected data can be guaranteed.
3. The shopping guide platform big data processing system according to claim 1, wherein the data preprocessing module comprises a MapReduce distributed computing framework and a central processing unit, the MapReduce distributed computing framework is a batch processing distributed computing framework, massive data are analyzed and processed by the central processing unit, and meanwhile, various structured data modules and unstructured data modules are processed, so that the data reading and writing and moving expenses are effectively reduced, and the processing and operating speed of big data is increased.
4. The shopping guide platform big data processing system as claimed in claim 1, wherein the data analysis module includes a data access connector module, an ETHINK data mining module, a network transmission module and a data visualization module, the data access connector module can identify the authority of the control end to data processing, and the ETHINK data mining module performs deep mining on big data and relational data through functions by means of the data access connector module and transmits the data to the data visualization module through the network transmission module.
5. The shopping guide platform big data processing system according to claim 4, wherein the data visualization module specifically comprises a tablet, a mobile client and a PC.
6. The shopping guide platform big data processing system as claimed in claim 1, wherein the shopping guide platform module is signally connected with a big data privacy and security module, the big data privacy and security module specifically comprises an IDS control authentication module signally connected with a firewall module, the firewall module is connected with a data entry module, and data is transferred to the website data backup module for backup through the data entry module.
7. The shopping guide platform big data processing system as claimed in claim 1, wherein the shopping guide platform module is connected with a big data application module and a data source collection module through signals, the data source collection module is transmitted to the signal converter module through the wireless network transmission module, the optical signals are converted into electrical signals through the signal converter module, and the electrical signals are transmitted to the mobile client module, so that the client can conveniently view data.
CN201911033638.4A 2019-10-28 2019-10-28 Shopping guide platform big data processing system Withdrawn CN110781362A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911033638.4A CN110781362A (en) 2019-10-28 2019-10-28 Shopping guide platform big data processing system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911033638.4A CN110781362A (en) 2019-10-28 2019-10-28 Shopping guide platform big data processing system

Publications (1)

Publication Number Publication Date
CN110781362A true CN110781362A (en) 2020-02-11

Family

ID=69387297

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911033638.4A Withdrawn CN110781362A (en) 2019-10-28 2019-10-28 Shopping guide platform big data processing system

Country Status (1)

Country Link
CN (1) CN110781362A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111878171A (en) * 2020-07-31 2020-11-03 谭静 Mine geological survey information system

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111878171A (en) * 2020-07-31 2020-11-03 谭静 Mine geological survey information system

Similar Documents

Publication Publication Date Title
CN101957834A (en) Content recommending method and device based on user characteristics
EP3321821A1 (en) Big data exchange method and device
Zeufack et al. An unsupervised anomaly detection framework for detecting anomalies in real time through network system’s log files analysis
CN110781362A (en) Shopping guide platform big data processing system
CN115640707A (en) Tunneling monitoring system and method, electronic equipment and storage medium
CN115118777A (en) Message conversion method, device, equipment and storage medium based on service type
CN110955890A (en) Method and device for detecting malicious batch access behaviors and computer storage medium
CN112565062A (en) Processing method, related device, equipment and medium of instant messaging order
CN115567563B (en) Comprehensive transportation hub monitoring and early warning system based on end edge cloud and control method thereof
CN109218342A (en) Equipment operating data acquisition processing system based on intelligence sensor
CN111711637A (en) Network communication technology's promotion safety guarantee system
CN203219505U (en) WiFi hand-held terminal used in safety production cloud service platform oriented to industrial and mining enterprises
CN115022722A (en) Video monitoring method and device, electronic equipment and storage medium
CN109785054A (en) The processing method of remote customisation converted products
CN110569424A (en) Information recommendation method and device
CN114298805A (en) Alarm information generation method and equipment
CN212009835U (en) Traffic big data analysis early warning platform
CN101692721A (en) Method for obtaining coal-mine gas monitoring data based on short messages
CN102882697A (en) Message receiving method of multi-client end of network management system based on callback system
CN113837281A (en) Metallurgical plant Internet platform and data regeneration method
CN112532512A (en) Session message processing method, related device, equipment and medium
CN110995650A (en) Multi-terminal checking early warning system based on industrial control base line
CN205318489U (en) Real -time supervision equipment of engineering equipment
CN110766498A (en) Method and device for recommending commodities
CN216751819U (en) Intelligent detection system based on 5G communication technology

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20200211

WW01 Invention patent application withdrawn after publication