CN112508759B - Big data management and mining system based on micro-service - Google Patents

Big data management and mining system based on micro-service Download PDF

Info

Publication number
CN112508759B
CN112508759B CN202011436214.5A CN202011436214A CN112508759B CN 112508759 B CN112508759 B CN 112508759B CN 202011436214 A CN202011436214 A CN 202011436214A CN 112508759 B CN112508759 B CN 112508759B
Authority
CN
China
Prior art keywords
questionnaire
unit
data
module
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011436214.5A
Other languages
Chinese (zh)
Other versions
CN112508759A (en
Inventor
刘谢慧
黄冬虹
赵彤
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qingyan Lingzhi Information Consulting Beijing Co ltd
Original Assignee
Qingyan Lingzhi Information Consulting Beijing Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qingyan Lingzhi Information Consulting Beijing Co ltd filed Critical Qingyan Lingzhi Information Consulting Beijing Co ltd
Priority to CN202011436214.5A priority Critical patent/CN112508759B/en
Publication of CN112508759A publication Critical patent/CN112508759A/en
Application granted granted Critical
Publication of CN112508759B publication Critical patent/CN112508759B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Systems or methods specially adapted for specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services
    • G06Q50/26—Government or public services
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/248—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/26—Visual data mining; Browsing structured data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0637—Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
    • G06Q10/06375—Prediction of business process outcome or impact based on a proposed change

Abstract

The invention provides a big data management and mining system based on micro service, comprising: the gateway module, the micro-service module and the database module; the gateway module is used for establishing communication connection with the user terminal; the user terminal is used for accessing the micro-service module through the gateway module or transmitting the collected questionnaire data to the micro-service module through the gateway module; the micro-service module comprises a data management unit and an analysis processing unit; the data management unit is used for preprocessing the acquired questionnaire data and classifying and inputting the preprocessed questionnaire data into the database module; the analysis processing unit is used for calling a big data analysis engine, mining, analyzing and processing questionnaire data stored in the database module, and outputting a corresponding big data analysis result to the user terminal; and the database module is used for classifying and storing the questionnaire data. The invention is beneficial to improving the comprehensive utilization level of research service resources.

Description

Big data management and mining system based on micro-service
Technical Field
The invention relates to the technical field of micro-services, in particular to a micro-service-based big data management and mining system.
Background
Currently, for research service development of government organs and departments, research staff mostly need to visit a base layer to develop research, collect research questionnaires filled by interviewees in the research process, and count filling information in the research questionnaires and count related research results according to the collected research questionnaires.
However, currently, the acquired questionnaire data cannot be reasonably utilized, and resources are wasted.
Disclosure of Invention
In view of the above problems, the present invention is directed to providing a microservice-based big data management and mining system.
The purpose of the invention is realized by adopting the following technical scheme:
the invention shows a microservice-based big data management and mining system, comprising: the gateway module, the micro-service module and the database module;
the gateway module is used for establishing communication connection with the user terminal; the user terminal is used for accessing the micro-service module through the gateway module or transmitting the collected questionnaire data to the micro-service module through the gateway module;
the micro-service module comprises a data management unit and an analysis processing unit; the data management unit is used for preprocessing the acquired questionnaire data and classifying and inputting the preprocessed questionnaire data into the database module; the analysis processing unit is used for calling a big data analysis engine, mining, analyzing and processing questionnaire data stored in the database module, and outputting a corresponding big data analysis result to the user terminal;
and the database module is used for classifying and storing the questionnaire data.
Optionally, the gateway module includes a verification unit;
the verification unit is used for verifying the access request of the user terminal, and allowing the user terminal to access the micro-service module through the gateway module after the verification is passed.
Optionally, the user terminal includes an access terminal and an information entry terminal;
the access terminal is used for accessing the analysis processing unit in the micro-service module and acquiring a corresponding big data analysis result by calling the analysis processing unit;
the information input terminal is used for collecting the questionnaire data, sending the collected questionnaire data to the data management unit in the micro-service module, and the data management unit preprocesses the questionnaire data and then inputs the questionnaire data into the database module in a classified mode.
Optionally, the data management unit includes a data cleaning unit and a data screening unit;
the data cleaning unit is used for carrying out error value correction, missing value processing, repeated data deletion and data formatting on the acquired questionnaire data;
the data screening unit is used for carrying out classification and calibration processing on the questionnaire data.
Optionally, the analysis processing unit includes a data mining unit, an orientation analysis processing unit, and a visualization unit;
the data mining unit is used for calling a related statistical algorithm model, a machine learning algorithm model and a deep learning algorithm model to perform data mining processing on questionnaire data stored in the database module according to a set data mining flow and outputting a data mining result;
the directional analysis unit is used for calling a corresponding analysis model according to target parameters set by a user, inputting corresponding classification label data serving as the analysis model, and outputting a specified analysis result through the analysis model;
and the visualization unit is used for visually displaying the results output by the data mining unit and the directional analysis unit.
Optionally, the database module includes a questionnaire database, an algorithm database and a user database;
the questionnaire database is used for classifying and storing the acquired questionnaire data;
the algorithm database is used for storing the statistical algorithm model, the machine learning algorithm model, the deep learning algorithm model and the analysis model;
the user database is used for storing user information of the system and authority information corresponding to the user, and storing equipment information of the information input terminal.
Optionally, the information entry terminal includes a communication unit, and an interaction unit and/or a scanning unit;
the interaction unit is used for displaying the electronic questionnaire and allowing the investigation object to fill in the electronic questionnaire through the interaction unit and then obtain a corresponding electronic document of the questionnaire;
the scanning unit is used for scanning the paper questionnaire filled in the research object to acquire a corresponding questionnaire image
The communication unit is in communication connection with the gateway module and is used for sending the acquired electronic documents and/or images of the questionnaires to the data management unit through the gateway module.
The invention has the beneficial effects that: the micro-service-based big data management and mining system provided by the invention can effectively carry out unified management on the questionnaire data, construct a unified questionnaire database, and is also beneficial to secondary big data mining and analysis processing of managers aiming at a large amount of questionnaire data in the database to obtain further research results, thereby improving the comprehensive utilization level of research service resources.
Drawings
The invention is further illustrated by means of the attached drawings, but the embodiments in the drawings do not constitute any limitation to the invention, and for a person skilled in the art, other drawings can be obtained on the basis of the following drawings without inventive effort.
FIG. 1 is a block diagram of a framework for a microservice-based big data management and mining system according to an embodiment of the present invention;
FIG. 2 is a block diagram of a framework of a microservice-based big data management and mining system according to another embodiment of the present invention.
Detailed Description
The invention is further described in connection with the following application scenarios.
Referring to fig. 1, the embodiment of the invention shows a microservice-based big data management and mining system, which comprises: the gateway module, the micro-service module and the database module;
the gateway module is used for establishing communication connection with the user terminal; the user terminal is used for accessing the micro-service module through the gateway module or transmitting the collected questionnaire data to the micro-service module through the gateway module;
the micro-service module comprises a data management unit and an analysis processing unit; the data management unit is used for preprocessing the acquired questionnaire data and classifying and inputting the preprocessed questionnaire data into the database module; the analysis processing unit is used for calling a big data analysis engine, mining, analyzing and processing questionnaire data stored in the database module, and outputting a corresponding big data analysis result to the user terminal;
and the database module is used for classifying and storing the questionnaire data.
In the above embodiment, the big data management and mining system constructed based on the micro-service can effectively perform unified management on the questionnaire data, construct a unified questionnaire database, and simultaneously facilitate managers to perform secondary big data mining and analysis processing on a large amount of questionnaire data in the database to obtain further research results, thereby improving the comprehensive utilization level of research service resources.
The user terminal is used as an access terminal and can provide functions of user interaction and questionnaire data entry.
The micro-service module is established based on a micro-service architecture, each functional unit in the micro-service module can be used as one or more basic micro-services, and each micro-service independently or cooperatively completes a corresponding function.
The gateway module is used as a uniform access port of the micro-service module to perform uniform authentication and access processing on the accessed user terminal.
Optionally, the gateway module includes a verification unit;
the authentication unit is used for authenticating the access request of the user terminal, and when the authentication is passed or the user terminal is allowed to access the micro-service module through the gateway module.
Optionally, the user terminal includes an access terminal and an information entry terminal;
the access terminal is used for accessing the analysis processing unit in the micro-service module and acquiring a corresponding big data analysis result by calling the analysis processing unit;
the information input terminal is used for collecting the questionnaire data, sending the collected questionnaire data to the data management unit in the micro-service module, and the data management unit preprocesses the questionnaire data and then inputs the questionnaire data into the database module in a classified mode.
The user terminal can be divided into an access terminal and an information input terminal, wherein the access terminal can be used for a user to access a specific analysis processing unit in the system, and the corresponding analysis processing unit is called to carry out big data mining and analysis processing on questionnaire data stored in the database module to obtain a corresponding analysis result, so that the functional analysis processing of the system is realized. The information input terminal is an intelligent terminal, the connection relation is established with a data management unit in the system through set adjustment, the acquired questionnaire data can be automatically and directly sent to the corresponding data management unit, and the data management unit performs further classified storage processing according to the received questionnaire data, so that the intelligent establishment of the database module is formed.
Optionally, the data management unit includes a data cleaning unit and a data screening unit;
the data cleaning unit is used for carrying out processing such as error value correction, missing value processing, repeated data deletion, data formatting and the like on the acquired questionnaire data;
the data screening unit is used for carrying out classification and calibration processing on the questionnaire data.
Optionally, the analysis processing unit includes a data mining unit, an orientation analysis processing unit, and a visualization unit;
the data mining unit is used for calling related statistical algorithm, machine learning algorithm, deep learning algorithm and the like to carry out data mining processing on questionnaire data stored in the database module according to a set data mining flow and outputting a data mining result;
the directional analysis unit is used for calling a corresponding analysis model according to target parameters set by a user, inputting corresponding classification label data serving as the analysis model, and outputting a specified analysis result through the analysis model;
and the visualization unit is used for visually displaying the results output by the data mining unit and the directional analysis unit.
Optionally, the database module includes a questionnaire database, an algorithm database and a user database;
the questionnaire database is used for classifying and storing the acquired questionnaire data;
the algorithm database is used for storing a statistical algorithm model, a machine learning algorithm model, a deep learning algorithm model, an analysis model and the like;
the user database is used for storing user information of the system and authority information corresponding to the user, and storing equipment information of the information input terminal.
The questionnaire data entered therein may be in the form of an electronic document or a picture.
Optionally, the information entry terminal includes an interaction unit and a communication unit;
the interaction unit is used for displaying the electronic questionnaire and allowing the investigation object to fill in the electronic questionnaire through the interaction unit and then obtain a corresponding electronic document of the questionnaire;
the communication unit establishes communication connection with the gateway module and is used for sending the acquired electronic document of the research and study to the data management unit through the gateway module.
Optionally, the information entry terminal further includes a scanning unit;
the scanning unit is used for scanning the paper questionnaire filled in by the research object to acquire a corresponding questionnaire image;
the communication unit is also used for sending the acquired questionnaire image to the data management unit through the gateway module.
In the above embodiment, for the intelligent input of the questionnaire data, the information input terminal may be an intelligent terminal device, and the corresponding questionnaire is filled in the terminal device by the visited object by displaying the questionnaire of the electronic version, so as to obtain an electronic document of the questionnaire filled in by the visited object as questionnaire data; the information input terminal can also scan the paper-based questionnaire to obtain a questionnaire picture, and the questionnaire picture is input into the system as questionnaire data.
In an optional implementation manner, the microserver module is further provided with a recognition unit for specifically recognizing the information of the questionnaire image, so that the obtained questionnaire image is converted into the text information corresponding to the text, and the digitized questionnaire information is formed and stored in a classified manner.
Optionally, the information entry terminal includes a scan enhancing unit;
the scanning enhancement unit is used for preprocessing the acquired questionnaire image and inputting the preprocessed questionnaire image.
And the communication unit sends the preprocessed questionnaire image to the data management unit through the gateway module.
Optionally, the scan enhancing unit performs preprocessing on the obtained questionnaire image, including:
converting the obtained questionnaire image into a gray image;
aiming at the acquired gray level image, calculating the region density factor of each position in the image, wherein the adopted region density factor calculation function is as follows:
Figure BDA0002828869470000051
Figure BDA0002828869470000052
in the formula, γ (x, y) represents a regional density factor of a pixel (x, y), where Φ 2 represents a pixel set within a 5 × 5 range with the pixel (x, y) as a central pixel, β (i, j) represents a dense parameter of the pixel (i, j) in the pixel set Φ 2, where Φ 1 represents a pixel set within a 10 × 10 range with the pixel (i, j) as a central pixel, and σ represents a local density factor of the pixel (x, y)Φ1(i, j) represents the gray variance, σ, of the pixel points in the set Φ 1Φ1-max(A) Represents the maximum value of the gray variance of the pixel points within the range of 10 multiplied by 10 with each pixel point as the center in the gray image, h (i, j) represents the gray value of the pixel point (i, j), P5-max(A) Histogram statistics is performed on gray values of all pixel points in a gray image, and the continuous 5 gray levels h in the gray histogram are the gray level median h 'in the continuous 5 gray levels with the largest number of pixel points { h' -2, h '-1, h', h '+ 1, h' +2 }; omega1And ω2Respectively represent the set adjustment factors;
according to the obtained region intensity factor of each pixel point, the brightness enhancement processing is carried out on the gray level image, and the method comprises the following steps:
converting the gray level image into a Lab color space, and respectively obtaining a brightness component L, a color component a and a color component b of the image;
aiming at the acquired brightness component L, the brightness component value of each pixel point in the image is subjected to enhancement adjustment, and the brightness component after enhancement adjustment is acquired
Figure BDA0002828869470000053
The brightness enhancement adjustment function used is:
Figure BDA0002828869470000061
in the formula (I), the compound is shown in the specification,
Figure BDA0002828869470000062
expressing the brightness component value of the pixel point (x, y) after the enhancement adjustment, and expressing the area concentration factor of the pixel point (x, y) by gamma (x, y), and expressing the brightness component value of the pixel point (x, y) after the enhancement adjustmentTIndicating a set region density factor threshold, LbIndicating a set first reference brightness value, wherein Lb∈[65，70]L (x, y) represents a luminance component value of the pixel point (x, y), LcIndicating a set second reference brightness value, wherein Lc∈[15，25]；
Adjusted luminance component according to enhancement
Figure BDA0002828869470000063
And reconstructing the color component a and the color component b, and outputting the preprocessed questionnaire image.
In the above embodiment, a special scan enhancing unit is provided in the information input terminal to pre-process the scanned questionnaire image, wherein an image enhancement processing technical scheme is provided, which is particularly directed to the characteristics of the questionnaire image, in which firstly, a region intensity factor calculation function is provided to reflect the information intensity condition of each region in the questionnaire image, then, adaptive brightness enhancement processing is performed for the position with higher information intensity, wherein the brightness component of the image in the Lab color space is taken as the basis in the enhancement processing process, and the brightness of the part with higher density is adjusted in a targeted manner so as to improve the definition of the part containing important information in the questionnaire, thereby improving the reliability and effect of data entry of the questionnaire.
In a preferred embodiment, the threshold value γ of the area density factorT＝μ×γmaxWhere μ represents a set threshold adjustment factor, μ ∈ [0.7, 0.9 ]]，γmaxAnd expressing the maximum value of the region density factor of each pixel point in the image.
It should be noted that, functional units/modules in the embodiments of the present invention may be integrated into one processing unit/module, or each unit/module may exist alone physically, or two or more units/modules are integrated into one unit/module. The integrated units/modules may be implemented in the form of hardware, or may be implemented in the form of software functional units/modules.
From the above description of embodiments, it is clear for a person skilled in the art that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code or any appropriate combination thereof. For a hardware implementation, a processor may be implemented in one or more of the following units: an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a processor, a controller, a microcontroller, a microprocessor, other electronic units designed to perform the functions described herein, or a combination thereof. For a software implementation, some or all of the procedures of an embodiment may be performed by a computer program instructing associated hardware. In practice, the program may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. Computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
Finally, it should be noted that the above embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the protection scope of the present invention, although the present invention is described in detail with reference to the preferred embodiments, it should be analyzed by those skilled in the art that modifications or equivalent substitutions can be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims (5)

1. Big data management and excavation system based on little service, characterized by that includes: the gateway module, the micro-service module and the database module;
the gateway module is used for establishing communication connection with the user terminal; the user terminal is used for accessing the micro-service module through the gateway module or transmitting the collected questionnaire data to the micro-service module through the gateway module;
the micro-service module comprises a data management unit and an analysis processing unit; the data management unit is used for preprocessing the acquired questionnaire data and classifying and inputting the preprocessed questionnaire data into the database module; the analysis processing unit is used for calling a big data analysis engine, mining, analyzing and processing questionnaire data stored in the database module, and outputting a corresponding big data analysis result to the user terminal;
the database module is used for classifying and storing the questionnaire data;
the gateway module comprises an authentication unit;
the verification unit is used for verifying the access request of the user terminal, and allowing the user terminal to access the micro service module through the gateway module after the verification is passed;
the user terminal comprises an access terminal and an information input terminal;
the access terminal is used for accessing the analysis processing unit in the micro service module and acquiring a corresponding big data analysis result by calling the analysis processing unit;
the information input terminal is used for collecting the questionnaire data, sending the collected questionnaire data to the data management unit in the micro-service module, preprocessing the questionnaire data by the data management unit and then inputting the preprocessed questionnaire data into the database module in a classified manner;
the information input terminal comprises a communication unit, a scanning unit and a scanning enhancement unit;
the scanning unit is used for scanning the paper questionnaire filled in by the research object to acquire a corresponding questionnaire image;
the scanning enhancement unit is used for preprocessing the acquired questionnaire image and outputting the preprocessed questionnaire image;
the communication unit is in communication connection with the gateway module and is used for sending the acquired preprocessed questionnaire image to the data management unit through the gateway module;
the micro-service module comprises an identification unit, wherein the identification unit is used for converting the acquired questionnaire images into corresponding character information so as to form digitalized questionnaire information for classified storage;
the scanning enhancement unit is used for preprocessing the acquired questionnaire image and comprises the following steps:
converting the obtained questionnaire image into a gray image;
aiming at the acquired gray level image, calculating the region density factor of each position in the image, wherein the adopted region density factor calculation function is as follows:
Figure FDA0003498468970000011
Figure FDA0003498468970000021
in the formula, γ (x, y) represents a regional density factor of a pixel (x, y), where Φ 2 represents a pixel set within a 5 × 5 range with the pixel (x, y) as a central pixel, β (i, j) represents a dense parameter of the pixel (i, j) in the pixel set Φ 2, where Φ 1 represents a pixel set within a 10 × 10 range with the pixel (i, j) as a central pixel, and σ represents a local density factor of the pixel (x, y)Φ1(i, j) represents the gray variance, σ, of the pixel points in the set Φ 1Φ1-max(A) Represents the maximum value of the gray variance of the pixel points within the range of 10 multiplied by 10 with each pixel point as the center in the gray image, h (i, j) represents the gray value of the pixel point (i, j), P5-max(A) Histogram statistics is performed on gray values of all pixel points in a gray image, and the continuous 5 gray levels h in the gray histogram are the gray level median h 'in the continuous 5 gray levels with the largest number of pixel points { h' -2, h '-1, h', h '+ 1, h' +2 }; omega1And ω2Respectively represent the set adjustment factors;
according to the obtained region intensity factor of each pixel point, the brightness enhancement processing is carried out on the gray level image, and the method comprises the following steps:
converting the gray level image into a Lab color space, and respectively obtaining a brightness component L, a color component a and a color component b of the image;
aiming at the acquired brightness component L, the brightness component value of each pixel point in the image is subjected to enhancement adjustment, and the brightness component after enhancement adjustment is acquired
Figure FDA0003498468970000024
The brightness enhancement adjustment function used is:
Figure FDA0003498468970000022
in the formula (I), the compound is shown in the specification,
Figure FDA0003498468970000025
expressing the brightness component value of the pixel point (x, y) after the enhancement adjustment, and expressing the area concentration factor of the pixel point (x, y) by gamma (x, y), and expressing the brightness component value of the pixel point (x, y) after the enhancement adjustmentTIndicating a set region density factor threshold, LbIndicating a set first reference brightness value, wherein Lb∈[65,70]L (x, y) represents a luminance component value of the pixel point (x, y), LcIndicating a set second reference brightness value, wherein Lc∈[15,25]；
Adjusted luminance component according to enhancement
Figure FDA0003498468970000023
And reconstructing the color component a and the color component b, and outputting the preprocessed questionnaire image.
2. The microservice-based big data management and mining system of claim 1, wherein the data management unit comprises a data cleansing unit and a data screening unit;
the data cleaning unit is used for carrying out error value correction, missing value processing, repeated data deletion and data formatting on the acquired questionnaire data;
the data screening unit is used for carrying out classification and calibration processing on the questionnaire data.
3. The microservice-based big data management and mining system of claim 2, wherein the analytics processing unit comprises a data mining unit, a directed analytics processing unit, and a visualization unit;
the data mining unit is used for calling a related statistical algorithm model and a deep learning algorithm model to perform data mining processing on the questionnaire data stored in the database module according to a set data mining flow and outputting a data mining result;
the directional analysis unit is used for calling a corresponding analysis model according to target parameters set by a user, inputting corresponding classification label data serving as the analysis model, and outputting a specified analysis result through the analysis model;
and the visualization unit is used for visually displaying the results output by the data mining unit and the directional analysis unit.
4. The microservice-based big data management and mining system of claim 3, wherein the database module comprises a questionnaire database, an algorithm database, and a user database;
the questionnaire database is used for classifying and storing the questionnaire data;
the algorithm database is used for storing the statistical algorithm model, the machine learning algorithm model, the deep learning algorithm model and the analysis model;
the user database is used for storing user information of the system and authority information corresponding to the user, and storing equipment information of the information input terminal.
5. The microservice-based big data management and mining system according to claim 1, wherein the information entry terminal further comprises an interaction unit
The interaction unit is used for displaying the electronic questionnaire and allowing the investigation object to fill in the electronic questionnaire through the interaction unit and then obtain a corresponding electronic document of the questionnaire;
the communication unit is used for sending the acquired electronic document of the research to the data management unit through the gateway module.
CN202011436214.5A 2020-12-10 2020-12-10 Big data management and mining system based on micro-service Active CN112508759B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011436214.5A CN112508759B (en) 2020-12-10 2020-12-10 Big data management and mining system based on micro-service

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011436214.5A CN112508759B (en) 2020-12-10 2020-12-10 Big data management and mining system based on micro-service

Publications (2)

Publication Number Publication Date
CN112508759A CN112508759A (en) 2021-03-16
CN112508759B true CN112508759B (en) 2022-03-15

Family

ID=74970448

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011436214.5A Active CN112508759B (en) 2020-12-10 2020-12-10 Big data management and mining system based on micro-service

Country Status (1)

Country Link
CN (1) CN112508759B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113472640B (en) * 2021-09-02 2021-12-14 广州市威士丹利智能科技有限公司 Intelligent gateway information processing method and system
CN114821440B (en) * 2022-05-12 2023-03-24 清研灵智信息咨询(北京)有限公司 Mobile video stream content identification and analysis method based on deep learning
CN114708138B (en) * 2022-06-06 2022-10-04 天津联想协同科技有限公司 Network disk image watermark adding method and device, network disk and storage medium

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8027822B2 (en) * 2005-06-20 2011-09-27 Virgin Healthmiles, Inc. Interactive, internet supported health and fitness management system
US20080162209A1 (en) * 2006-12-28 2008-07-03 Oracle International Corporation Configurable actions in a dashboard application
CN201583999U (en) * 2009-12-10 2010-09-15 周江红 Distribution-type automatic survey questionnaire system
US9652776B2 (en) * 2012-06-18 2017-05-16 Greg Olsen Visual representations of recurring revenue management system data and predictions
CN103310211B (en) * 2013-04-26 2016-04-06 四川大学 A kind ofly fill in mark recognition method based on image procossing
IL281000B2 (en) * 2016-05-29 2023-12-01 Wix Com Ltd System and method for the creation and update of hierarchical websites based on collected business knowledge
KR101883332B1 (en) * 2016-06-07 2018-07-30 주식회사 건강한친구들 Service providing apparatus and method for personal training adapted to user
CN106708931A (en) * 2016-11-16 2017-05-24 相交科技（武汉）有限公司 Personal reading material production method
US10163006B2 (en) * 2017-02-27 2018-12-25 International Business Machines Corporation Selection determination for freehand marks
CN107679634A (en) * 2017-10-27 2018-02-09 国网陕西省电力公司西安供电公司 A kind of method that power supply trouble based on data visualization reports analysis and prediction for repairment
CN109977936A (en) * 2019-03-15 2019-07-05 深圳壹账通智能科技有限公司 Paper questionnaire statistical method, device, medium and computer equipment
CN110909050A (en) * 2019-12-04 2020-03-24 深圳瑞力网科技有限公司 Data statistical analysis system
CN111428446B (en) * 2020-02-26 2023-09-01 佛山四面体科技有限公司 Questionnaire identification method and questionnaire identification system

Also Published As

Publication number Publication date
CN112508759A (en) 2021-03-16

Similar Documents

Publication Publication Date Title
CN112508759B (en) Big data management and mining system based on micro-service
CN112434880B (en) Patrol route planning and patrol personnel management system based on deep learning
CN103975342A (en) Systems and methods for mobile image capture and processing
CN109871845B (en) Certificate image extraction method and terminal equipment
TW202026948A (en) Methods and devices for biological testing and storage medium thereof
US10346980B2 (en) System and method of processing medical images
WO2022105019A1 (en) Snapshot quality evaluation method and apparatus for vehicle bayonet device, and readable medium
CN112598670B (en) Land supervision platform based on cloud computing
CN113160023A (en) Land utilization checking system
CN112529752B (en) Trusty investigation platform based on block chain
US11817215B2 (en) Artificial intelligence cloud diagnosis platform
CN110543811A (en) non-cooperation type examination person management method and system based on deep learning
CN113590906A (en) Land utilization change proof checking system
CN115633193B (en) Monitoring video encryption method and system
CN110111165A (en) True from false of bills checking method, system, medium and electronic equipment
CN111445058A (en) Data analysis method, device, equipment and computer readable storage medium
CN114936981A (en) Code registration system is swept in place based on cloud platform
CN110647824A (en) Value-added tax invoice layout extraction method based on computer vision technology
CN108052918A (en) A kind of person's handwriting Compare System and method
CN101151882A (en) System and method of processing scan data
CN115546845B (en) Multi-view cow face recognition method and device, computer equipment and storage medium
CN116052090A (en) Image quality evaluation method, model training method, device, equipment and medium
CN115019376A (en) Cloud service-based power information management system
CN112598630A (en) Pulmonary nodule detection method, apparatus, medium, and computer program product
CN112651706A (en) Personal social security service handling system based on cloud computing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant