CN116483898A - Big data application development platform for multi-source data - Google Patents
Big data application development platform for multi-source data Download PDFInfo
- Publication number
- CN116483898A CN116483898A CN202310309614.7A CN202310309614A CN116483898A CN 116483898 A CN116483898 A CN 116483898A CN 202310309614 A CN202310309614 A CN 202310309614A CN 116483898 A CN116483898 A CN 116483898A
- Authority
- CN
- China
- Prior art keywords
- data
- sub
- analysis
- matching
- source data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011161 development Methods 0.000 title claims abstract description 41
- 238000004458 analytical method Methods 0.000 claims abstract description 137
- 238000007405 data analysis Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 12
- 230000000007 visual effect Effects 0.000 claims description 38
- 238000012795 verification Methods 0.000 claims description 35
- 238000013475 authorization Methods 0.000 claims description 12
- 238000011156 evaluation Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000012797 qualification Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 8
- 238000000034 method Methods 0.000 abstract description 4
- 230000009286 beneficial effect Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000010076 replication Effects 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a big data application development platform facing multi-source data, which comprises: the data receiving module is used for receiving multi-source data uploaded by the client based on the target receiving end; the learning module is used for inputting the multi-source data into a big data network for learning to obtain a target decision; the data analysis module is used for analyzing the multisource data uploaded by the client based on the target decision to generate an analysis report, and meanwhile, the analysis report is qualified and verified; and the storage module is used for visually storing the multi-source data uploaded by the client and the analysis report when the analysis report is qualified. The method is convenient for accurately and reliably managing the multi-source data, and the data with potential value is mined, so that the utilization rate of the multi-source data is improved, meanwhile, the multi-source data is analyzed and processed through the big data application and development platform, the efficiency and the accuracy of the multi-source data processing are improved, and the management effect of the multi-source data is guaranteed.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a big data application development platform for multi-source data.
Background
At present, the big data application development platform is a platform for analyzing mass data by means of big data technology, and can be used for efficiently processing the mass data, mining valuable data in the mass data, and being convenient for reasonably managing multi-source data;
however, in the prior art, the data development platform can only process one type of data at the same time, so that the data is processed singly and has low utilization rate, the processing efficiency of the data is further reduced, and the management of different data is not facilitated;
therefore, in order to overcome the problems, the invention provides a big data application development platform for multi-source data.
Disclosure of Invention
The invention provides a big data application development platform for multi-source data, which is used for analyzing and integrating the multi-source data through the big data, so that the multi-source data can be accurately and reliably managed, and the data with potential value can be mined, thereby being convenient for improving the utilization rate of the multi-source data, and simultaneously, the big data application development platform is used for analyzing and processing the multi-source data, improving the efficiency and accuracy of the multi-source data processing, and guaranteeing the management effect of the multi-source data.
The invention provides a big data application development platform facing multi-source data, which comprises:
the data receiving module is used for receiving multi-source data uploaded by the client based on the target receiving end;
the learning module is used for inputting the multi-source data into a big data network for learning to obtain a target decision;
the data analysis module is used for analyzing the multisource data uploaded by the client based on the target decision to generate an analysis report, and meanwhile, the analysis report is qualified and verified;
and the storage module is used for visually storing the multi-source data uploaded by the client and the analysis report when the analysis report is qualified.
Preferably, a big data application development platform facing to multi-source data, a data receiving module, includes:
the request generation unit is used for generating a data uploading request based on the client, wherein the data uploading request comprises a client address and a data type of multi-source data to be uploaded;
the verification unit is used for carrying out authorization verification in the target receiving end based on the address of the client and the data type of the multi-source data to be uploaded;
and the data uploading unit is used for receiving the multi-source data uploaded by the client based on the target receiving end when the verification is passed.
Preferably, a big data application development platform facing to multi-source data, a verification unit, includes:
the request processing subunit is used for acquiring the access rights of the information management library in the target receiving end, determining a request input standard based on the access rights, converting the data uploading request based on the request input standard to obtain a target data uploading request, and determining the request type of the target data uploading request;
the matching subunit is used for inputting a target data uploading request into the information management library based on the request type to perform first matching, and obtaining an authorized client address set and an authorized data type set which are consistent with the request type;
and the verification subunit is used for performing second matching on the client address in the target data uploading request and the authorized client address set, performing third matching on the data type of the multi-source data to be uploaded in the target data uploading request and the authorized data type set, and determining whether the target data uploading request passes the authorization verification or not based on the second matching and the third matching.
Preferably, a big data application development platform for multi-source data, the verification subunit includes:
and acquiring a matching result of the second matching, when the authorized client address is concentrated and has an authorized client address consistent with the client address, acquiring a matching result of the third matching, and when the authorized data type is concentrated and has an authorized data type consistent with the data type of the multi-source data to be uploaded, judging that the authorization verification is passed, otherwise, judging that the verification is not passed.
Preferably, a big data application development platform facing to multi-source data, a learning module, includes:
the classifying unit is used for classifying the multi-source data according to the data dimension to generate a plurality of sub-data segments, determining a segment identifier corresponding to each sub-data segment, searching the segment identifier corresponding to the sub-data segment with each network node in the big data network, and determining a target network node corresponding to each sub-data segment;
the evaluation unit is used for reading the strategy information data stored in the target network node, determining the strategy characteristics corresponding to the strategy information data, simultaneously acquiring the data structure of the sub-data segment corresponding to the strategy information data, performing fourth matching on the strategy characteristics and the data structure, and evaluating the executable degree of analysis on the corresponding sub-data segment by the strategy information data based on the fourth matching result;
a judging unit configured to:
acquiring an executable degree threshold, comparing the executable degree threshold with the executable degree of the analysis of the corresponding sub-data segment by the strategy information data, and judging whether the strategy information data is qualified or not;
when the executable degree of the strategy information data for analyzing the corresponding sub-data segment is equal to or greater than the executable degree threshold, judging that the strategy information data is qualified, and determining an analysis strategy for analyzing the corresponding sub-data segment based on the strategy information data;
otherwise, judging that the strategy information data is unqualified, and simultaneously, re-matching the target network node in the big data network.
Preferably, a big data application development platform facing to multi-source data, an evaluation unit, includes:
the reading subunit is used for reading the data structure, determining the data type, the total data amount and the data logic corresponding to the sub-data segment, simultaneously reading the strategy characteristics, determining the strategy analysis type, the total analysis amount and the strategy analysis logic corresponding to the strategy information data, and performing fourth matching on the data structure and the strategy information data;
a fourth matching unit for:
performing first sub-matching on the data type corresponding to the sub-data segment and the strategy analysis type corresponding to the strategy information data to obtain a first association degree of the sub-data segment and the strategy information data, and determining a first matching weight corresponding to the first sub-matching;
performing second sub-matching on the total data amount corresponding to the sub-data segment and the analysis total amount corresponding to the strategy information data to obtain a second association degree of the sub-data segment and the strategy information data, and determining a second matching weight corresponding to the second sub-matching;
performing third sub-matching on the data logic corresponding to the sub-data segment and the strategy analysis logic corresponding to the strategy information data to obtain a third association degree of the sub-data segment and the strategy information data, and determining a third matching weight corresponding to the third sub-matching;
and the calculating subunit is used for calculating the first association degree, the first matching weight, the second association degree, the second matching weight, the third association degree and the third matching weight, and determining the executable degree of analyzing the corresponding sub-data segment by the strategy information data based on the calculation result.
Preferably, a big data application development platform facing to multi-source data, a data analysis module, includes:
the data analysis unit is used for analyzing the multi-source data based on the target decision to obtain an analysis result of analyzing the multi-source data;
and the report generating unit is used for generating an analysis report based on the target decision, the multi-source data and the analysis result.
Preferably, a big data application development platform facing multi-source data, a report generating unit, includes:
the data attribute acquisition subunit is used for acquiring the data attribute of the multi-source data, calling a first file list in a preset document management library based on the data attribute of the multi-source data, automatically inputting the multi-source data into the first file list, and acquiring a first report;
the first copying subunit is used for carrying out first copying on the first report, generating a second file list based on the first copying result adding strategy column, and automatically inputting the target strategy into the second file list to obtain a second report;
and the second copying subunit is used for carrying out second copying on the second report, generating a third file list based on the second copying result adding result column, and automatically inputting the analysis result into the third file list to generate a third report, wherein the third report is an analysis report.
Preferably, a big data application development platform facing to multi-source data, a data analysis module, includes:
an identification unit for determining an analysis result of analyzing the multi-source data based on the analysis report;
the data reading unit is used for reading the multi-source data and determining an analysis result range for analyzing the multi-source data according to the data characteristics of the multi-source data;
a qualification verification unit, configured to:
comparing the analysis result with the analysis result range, and judging whether the analysis report is qualified or not;
when the analysis result is within the analysis result range, judging that the analysis report is qualified;
otherwise, judging that the analysis report is unqualified.
Preferably, a big data application development platform for multi-source data, a storage module, includes:
the storage visual window establishing unit is used for acquiring the data characteristics of the multi-source data and establishing a storage visual window according to the data characteristics of the multi-source data;
the storage visual sub-window establishing unit is used for correspondingly adding the storage visual sub-window according to the visual storage window;
and the visual storage unit is used for storing the multi-source data into the storage visual window, storing the analysis report into the storage visual sub-window, and completing visual storage of the multi-source data and the analysis report uploaded by the client based on the storage visual window and the storage visual sub-window.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a block diagram of a big data application development platform for multi-source data in an embodiment of the invention;
FIG. 2 is a block diagram of a data receiving module in a big data application development platform for multi-source data according to an embodiment of the present invention;
fig. 3 is a block diagram of a learning module in a big data application development platform for multi-source data according to an embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings, it being understood that the preferred embodiments described herein are for illustration and explanation of the present invention only, and are not intended to limit the present invention.
Example 1:
the embodiment provides a big data application development platform for multi-source data, as shown in fig. 1, including:
the data receiving module is used for receiving multi-source data uploaded by the client based on the target receiving end;
the learning module is used for inputting the multi-source data into a big data network for learning to obtain a target decision;
the data analysis module is used for analyzing the multisource data uploaded by the client based on the target decision to generate an analysis report, and meanwhile, the analysis report is qualified and verified;
and the storage module is used for visually storing the multi-source data uploaded by the client and the analysis report when the analysis report is qualified.
In this embodiment, the target receiving end may be an intelligent device capable of analyzing and processing multi-source data, and may specifically be a computer or the like.
In this embodiment, the client may be an intelligent terminal with data uploading and generating data to be analyzed, and may specifically be a device such as a mobile phone.
In this embodiment, the multi-source data may be data from a variety of different sources, and the data types are not unique.
In this embodiment, the target decision may be a data analysis method corresponding to different source data obtained from the big data network, and the same source data corresponds to a decision.
In this embodiment, the analysis report is used to record the analysis result of the multi-source data, so as to facilitate effective management of the multi-source data.
In this embodiment, the visual storage may store the analysis report and the multi-source data uploaded by the client, and the storage result is visible to the user, so that the user may retrieve and review the stored analysis report and multi-source data according to the requirement of self-review.
The beneficial effects of the technical scheme are as follows: the multi-source data is analyzed and integrated through the big data, so that the multi-source data can be accurately and reliably managed, the data with potential value is mined, the utilization rate of the multi-source data is improved, meanwhile, the multi-source data is analyzed and processed through the big data application development platform, the efficiency and the accuracy of the multi-source data processing are improved, and the management effect of the multi-source data is guaranteed.
Example 2:
on the basis of embodiment 1, this embodiment provides a big data application development platform for multi-source data, as shown in fig. 2, the data receiving module includes:
the request generation unit is used for generating a data uploading request based on the client, wherein the data uploading request comprises a client address and a data type of multi-source data to be uploaded;
the verification unit is used for carrying out authorization verification in the target receiving end based on the address of the client and the data type of the multi-source data to be uploaded;
and the data uploading unit is used for receiving the multi-source data uploaded by the client based on the target receiving end when the verification is passed.
In this embodiment, the data upload request is generated by the client for communicating data upload information to the target recipient.
In this embodiment, the authorization verification may be to verify the data uploading authority of the client through the target receiving end, so as to facilitate ensuring the security and reliability of the data.
The beneficial effects of the technical scheme are as follows: the data uploading request of the client is determined and analyzed through the target receiving end, so that the multi-source data of the client can be conveniently uploaded to the target receiving end after the authorization verification of the client is passed, the reliability of data uploading is guaranteed, and the guarantee is provided for analyzing and processing the multi-source data through the big data application development platform.
Example 3:
on the basis of embodiment 2, this embodiment provides a big data application development platform for multi-source data, and the verification unit includes:
the request processing subunit is used for acquiring the access rights of the information management library in the target receiving end, determining a request input standard based on the access rights, converting the data uploading request based on the request input standard to obtain a target data uploading request, and determining the request type of the target data uploading request;
the matching subunit is used for inputting a target data uploading request into the information management library based on the request type to perform first matching, and obtaining an authorized client address set and an authorized data type set which are consistent with the request type;
and the verification subunit is used for performing second matching on the client address in the target data uploading request and the authorized client address set, performing third matching on the data type of the multi-source data to be uploaded in the target data uploading request and the authorized data type set, and determining whether the target data uploading request passes the authorization verification or not based on the second matching and the third matching.
In this embodiment, the information management library is set in advance, and is used to store access right information corresponding to different clients.
In this embodiment, the requested data standard may be a data format requirement and an uploading manner characterizing when uploading multi-source data to the target receiving end by different clients.
In this embodiment, the target data upload request may be specific data upload information obtained by analyzing the data upload request according to the request input standard, including the upload mode, the type, and the like.
In this embodiment, the first matching may be matching the obtained target data upload request with the access rights of each client stored in the information management library.
In this embodiment, the authorized client address set may be a set of all client addresses in the information management library that are consistent with the client's target data upload request.
In this embodiment, the authorized data type set may be all the data type sets that can be uploaded in the information management library in accordance with the target data upload request.
In this embodiment, the second matching may be to match the client address in the target data upload request with the obtained authorized client address set, so as to implement authority verification on the client address of the client.
In this embodiment, the third matching may be to match the data type of the multi-source data to be uploaded with the authorized data type set, so as to verify the uploading authority of the data type.
The beneficial effects of the technical scheme are as follows: by analyzing the data uploading request of the client, the authority verification is respectively carried out on the address of the client and the data type of the data to be uploaded contained in the data uploading request, so that the client is ensured to have the authority of uploading the data, the reliability and the safety of uploading the multi-source data are ensured, and convenience and guarantee are provided for accurately and reliably analyzing the multi-source data.
Example 4:
on the basis of embodiment 3, this embodiment provides a big data application development platform for multi-source data, and the verification subunit includes:
and acquiring a matching result of the second matching, when the authorized client address is concentrated and has an authorized client address consistent with the client address, acquiring a matching result of the third matching, and when the authorized data type is concentrated and has an authorized data type consistent with the data type of the multi-source data to be uploaded, judging that the authorization verification is passed, otherwise, judging that the verification is not passed.
The beneficial effects of the technical scheme are as follows: the validity of the client is guaranteed by respectively carrying out authorization verification on the client address and the data type of the client, and the security is guaranteed for realizing high-efficiency analysis of the multi-source data through the big data application platform.
Example 5:
on the basis of embodiment 1, this embodiment provides a big data application development platform for multi-source data, as shown in fig. 3, the learning module includes:
the classifying unit is used for classifying the multi-source data according to the data dimension to generate a plurality of sub-data segments, determining a segment identifier corresponding to each sub-data segment, searching the segment identifier corresponding to the sub-data segment with each network node in the big data network, and determining a target network node corresponding to each sub-data segment;
the evaluation unit is used for reading the strategy information data stored in the target network node, determining the strategy characteristics corresponding to the strategy information data, simultaneously acquiring the data structure of the sub-data segment corresponding to the strategy information data, performing fourth matching on the strategy characteristics and the data structure, and evaluating the executable degree of analysis on the corresponding sub-data segment by the strategy information data based on the fourth matching result;
a judging unit configured to:
acquiring an executable degree threshold, comparing the executable degree threshold with the executable degree of the analysis of the corresponding sub-data segment by the strategy information data, and judging whether the strategy information data is qualified or not;
when the executable degree of the strategy information data for analyzing the corresponding sub-data segment is equal to or greater than the executable degree threshold, judging that the strategy information data is qualified, and determining an analysis strategy for analyzing the corresponding sub-data segment based on the strategy information data;
otherwise, judging that the strategy information data is unqualified, and simultaneously, re-matching the target network node in the big data network.
In this embodiment, the data dimensions are set in advance, and one data type corresponds to one dimension, so that the obtained multi-source data can be accurately and effectively classified.
In this embodiment, the sub-data segment may be a different type of data set obtained by classifying the obtained multi-source data.
In this embodiment, the segment identification may be a tag label that marks different sub-data segments, by which the sub-data segments can be quickly distinguished.
In this embodiment, the network node may be a tool for storing policy information data corresponding to different types of data in the big data network, and is not unique.
In this embodiment, the target network node may be a node that is included in the big data network and matches the segment identifier, and each sub-data segment corresponds to a target network node.
In this embodiment, the policy information data may be data such as rules and methods for analyzing multi-source data.
In this embodiment, the policy features may be key data that can characterize the type and characteristics of the different policy information data.
In this embodiment, the data structure may be a data type capable of characterizing the sub-data segment, an association relationship between each data in the sub-data segment, and the like.
In this embodiment, the fourth matching may be to match the policy features with the data structure, so as to achieve the executable degree of analyzing the policy information data on different sub-data segments.
In this embodiment, the executable degree is used to characterize the executable condition of the policy information data when analyzing the sub-data segment, and the larger the value, the more fit the policy information data and the sub-data segment is characterized.
In this embodiment, the threshold of the executable degree is set in advance, and is used for representing the lowest value of the executable degree, namely the lowest standard.
The beneficial effects of the technical scheme are as follows: classifying the obtained multi-source data through data dimension, determining segment identifiers corresponding to different sub-data segments obtained after classification, matching the large data network with a target network node, reading policy information data contained in the target network node, analyzing the executable degree of the sub-data segments according to the reading result, thereby ensuring the correspondence between the policy information data and the sub-data segments, and finally determining analysis strategies corresponding to different sub-data segments through the policy information data, so that the accurate and reliable analysis of the sub-data segments is realized, the efficiency and the accuracy of the multi-source data analysis through a large data application platform are ensured, and the effective management of the multi-source data is realized.
Example 6:
on the basis of embodiment 5, this embodiment provides a big data application development platform for multi-source data, and the evaluation unit includes:
the reading subunit is used for reading the data structure, determining the data type, the total data amount and the data logic corresponding to the sub-data segment, simultaneously reading the strategy characteristics, determining the strategy analysis type, the total analysis amount and the strategy analysis logic corresponding to the strategy information data, and performing fourth matching on the data structure and the strategy information data;
a fourth matching unit for:
performing first sub-matching on the data type corresponding to the sub-data segment and the strategy analysis type corresponding to the strategy information data to obtain a first association degree of the sub-data segment and the strategy information data, and determining a first matching weight corresponding to the first sub-matching;
performing second sub-matching on the total data amount corresponding to the sub-data segment and the analysis total amount corresponding to the strategy information data to obtain a second association degree of the sub-data segment and the strategy information data, and determining a second matching weight corresponding to the second sub-matching;
performing third sub-matching on the data logic corresponding to the sub-data segment and the strategy analysis logic corresponding to the strategy information data to obtain a third association degree of the sub-data segment and the strategy information data, and determining a third matching weight corresponding to the third sub-matching;
and the calculating subunit is used for calculating the first association degree, the first matching weight, the second association degree, the second matching weight, the third association degree and the third matching weight, and determining the executable degree of analyzing the corresponding sub-data segment by the strategy information data based on the calculation result.
In this embodiment, determining the executable degree of the analysis of the corresponding sub-data segment by the policy information data based on the calculation result includes:
calculating the executable degree of analyzing the corresponding sub-data segment according to the following formula;
δ＝ρ 1 *ω 1 +ρ 2 *ω 2 +ρ 3 *ω 3 ；
wherein delta represents the executable degree of analyzing the corresponding sub-data segment by the strategy information data; ρ 1 Representing a first degree of association; omega 1 Representing a first matching weight; ρ 2 Representing a second degree of association; omega 2 Representing a second matching weight; ρ 3 Representing a third degree of association; omega 3 Representing a third matching weight.
In this embodiment, the data logic may be an association relationship between each data in the sub-data segment.
In this embodiment, the policy analysis logic may be an order of data analysis, a logical consecutive order, and the like of the policy information data when analyzing the sub-data segments.
In this embodiment, the first sub-matching may be to match a data type corresponding to the sub-data segment with a policy analysis type corresponding to the policy information data, so as to verify a type adaptation degree between the sub-data segment and the policy analysis type.
In this embodiment, the first association degree is used to characterize the association degree of the type between the sub-data segment and the policy information data, and a larger value indicates that the sub-data segment and the policy information data are matched.
In this embodiment, the first matching weight may be a measure of importance characterizing the first sub-match in determining the degree of executability.
In this embodiment, the second sub-match may be that the total amount of data corresponding to the sub-data segment matches the total amount of analysis corresponding to the policy information data.
In this embodiment, the second association degree may be a degree of association for characterizing a data amount between the sub-data segment and the policy information data, where a larger value indicates a matching of the sub-data segment and the policy information data.
In this embodiment, the second match weight may be indicative of how important the second sub-match is in determining the degree of executability.
In this embodiment, the data logic corresponding to the third sub-match sub-data segment matches the policy analysis logic corresponding to the policy information data.
In this embodiment, the third association degree may be used to characterize the association degree of logic between the sub-data segment and the policy information data, where a larger value indicates a matching of the sub-data segment and the policy information data.
In this embodiment, the third matching weight may be indicative of how important the third sub-match is in determining the degree of executability.
The beneficial effects of the technical scheme are as follows: the data type, the total data amount, the strategy analysis type, the total analysis amount and the strategy analysis logic corresponding to the data segments, the data logic and the strategy information data are respectively determined, so that the data type, the total analysis amount and the strategy analysis logic are matched in a one-to-one correspondence manner, the executable degree of analyzing the corresponding data segments by the strategy information data is accurately and effectively evaluated, the accuracy and the efficiency of multi-source data analysis are ensured, and the management effect of the multi-source data by a big data application platform is improved.
Example 7:
on the basis of embodiment 1, this embodiment provides a big data application development platform for multi-source data, and a data analysis module, including:
the data analysis unit is used for analyzing the multi-source data based on the target decision to obtain an analysis result of analyzing the multi-source data;
and the report generating unit is used for generating an analysis report based on the target decision, the multi-source data and the analysis result.
The beneficial effects of the technical scheme are as follows: and finally, generating a corresponding analysis report by the analysis result, the target decision and the multi-source data, so that a user can conveniently and timely inquire, and effective management of the multi-source data is realized.
Example 8:
on the basis of embodiment 7, this embodiment provides a big data application development platform for multi-source data, and the report generating unit includes:
the data attribute acquisition subunit is used for acquiring the data attribute of the multi-source data, calling a first file list in a preset document management library based on the data attribute of the multi-source data, automatically inputting the multi-source data into the first file list, and acquiring a first report;
the first copying subunit is used for carrying out first copying on the first report, generating a second file list based on the first copying result adding strategy column, and automatically inputting the target strategy into the second file list to obtain a second report;
and the second copying subunit is used for carrying out second copying on the second report, generating a third file list based on the second copying result adding result column, and automatically inputting the analysis result into the third file list to generate a third report, wherein the third report is an analysis report.
In this embodiment, the data attributes may be key data pieces that can characterize the data type and amount of the multi-source data.
In this embodiment, the preset document management library is set in advance, and is used for storing different file templates.
In this embodiment, the first file list may be a file template with record multisource data.
In this embodiment, the first report may be an initial report obtained after the multi-source data is input into the first file list, where the report includes only multi-source data items.
In this embodiment, the first replication may be replication of the first report, thereby enabling the addition of a policy column (record target policy) on the basis of the first report to generate the second file list.
In this embodiment, the second file list may be a file template that records the target policy on the basis of the first file list.
In this embodiment, the second report may be a report containing the multi-source data and the target policy after the target policy is added to the second file list.
In this embodiment, the second copying may be to copy the second report, thereby implementing adding a result column (recording the analysis result) on the basis of the second report, and generating the third file list.
In this embodiment, the third file list may be a template for recording the analysis result on the basis of the second file list.
In this embodiment, the result column is used to record the analysis result of the target policy on the corresponding sub-data segment.
The beneficial effects of the technical scheme are as follows: the target decision, the multi-source data and the analysis result are recorded in the report template respectively, so that the analysis report is finally obtained, a user can accurately and reliably know the analysis condition of the multi-source data according to the analysis report, and the analysis effect of the big data application platform on the multi-source data is ensured.
Example 9:
on the basis of embodiment 1, this embodiment provides a big data application platform for multi-source data, and a data analysis module, including:
an identification unit for determining an analysis result of analyzing the multi-source data based on the analysis report;
the data reading unit is used for reading the multi-source data and determining an analysis result range for analyzing the multi-source data according to the data characteristics of the multi-source data;
a qualification verification unit, configured to:
comparing the analysis result with the analysis result range, and judging whether the analysis report is qualified or not;
when the analysis result is within the analysis result range, judging that the analysis report is qualified;
otherwise, judging that the analysis report is unqualified.
In this embodiment, the analysis result range may be a value interval corresponding to a theoretical analysis result after the analysis of the multi-source data by the analysis strategy is characterized.
The beneficial effects of the technical scheme are as follows: by comparing the analysis result of the multi-source data in the analysis report with the theoretical analysis result range corresponding to the multi-source data, whether the analysis report is qualified or not is accurately checked, so that the reasonability of the obtained analysis report is ensured, and the analysis accuracy and management effect of the multi-source data are improved.
Example 10:
on the basis of embodiment 1, this embodiment provides a big data application platform for multi-source data, and a storage module, including:
the storage visual window establishing unit is used for acquiring the data characteristics of the multi-source data and establishing a storage visual window according to the data characteristics of the multi-source data;
the storage visual sub-window establishing unit is used for correspondingly adding the storage visual sub-window according to the visual storage window;
and the visual storage unit is used for storing the multi-source data into the storage visual window, storing the analysis report into the storage visual sub-window, and completing visual storage of the multi-source data and the analysis report uploaded by the client based on the storage visual window and the storage visual sub-window.
In this embodiment, the data features may be the data type of the multi-source data, the value range of the multi-source data, and the like.
In this embodiment, the storage visual window may be a window that stores multi-source data.
In this embodiment, the store visual sub-window may be a window that stores analysis reports.
The beneficial effects of the technical scheme are as follows: by constructing the storage visual window according to the data characteristics of the multi-source data, the multi-source data and the analysis report are respectively stored through the storage visual window, the analysis result and the multi-source data are reliably stored, the user can conveniently and correspondingly inquire the data and the analysis result according to the self requirements, and the analysis effect and the management effect of the multi-source data are ensured.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. A big data application development platform for multi-source data, comprising:
the data receiving module is used for receiving multi-source data uploaded by the client based on the target receiving end;
the learning module is used for inputting the multi-source data into a big data network for learning to obtain a target decision;
the data analysis module is used for analyzing the multisource data uploaded by the client based on the target decision to generate an analysis report, and meanwhile, the analysis report is qualified and verified;
and the storage module is used for visually storing the multi-source data uploaded by the client and the analysis report when the analysis report is qualified.
2. The big data application development platform for multi-source data according to claim 1, wherein the data receiving module comprises:
the request generation unit is used for generating a data uploading request based on the client, wherein the data uploading request comprises a client address and a data type of multi-source data to be uploaded;
the verification unit is used for carrying out authorization verification in the target receiving end based on the address of the client and the data type of the multi-source data to be uploaded;
and the data uploading unit is used for receiving the multi-source data uploaded by the client based on the target receiving end when the verification is passed.
3. The big data application development platform for multi-source data according to claim 2, wherein the verification unit comprises:
the request processing subunit is used for acquiring the access rights of the information management library in the target receiving end, determining a request input standard based on the access rights, converting the data uploading request based on the request input standard to obtain a target data uploading request, and determining the request type of the target data uploading request;
the matching subunit is used for inputting a target data uploading request into the information management library based on the request type to perform first matching, and obtaining an authorized client address set and an authorized data type set which are consistent with the request type;
and the verification subunit is used for performing second matching on the client address in the target data uploading request and the authorized client address set, performing third matching on the data type of the multi-source data to be uploaded in the target data uploading request and the authorized data type set, and determining whether the target data uploading request passes the authorization verification or not based on the second matching and the third matching.
4. A multi-source data oriented big data application development platform in accordance with claim 3, wherein the verification subunit comprises:
and acquiring a matching result of the second matching, when the authorized client address is concentrated and has an authorized client address consistent with the client address, acquiring a matching result of the third matching, and when the authorized data type is concentrated and has an authorized data type consistent with the data type of the multi-source data to be uploaded, judging that the authorization verification is passed, otherwise, judging that the verification is not passed.
5. The big data application development platform for multi-source data according to claim 1, wherein the learning module comprises:
the classifying unit is used for classifying the multi-source data according to the data dimension to generate a plurality of sub-data segments, determining a segment identifier corresponding to each sub-data segment, searching the segment identifier corresponding to the sub-data segment with each network node in the big data network, and determining a target network node corresponding to each sub-data segment;
the evaluation unit is used for reading the strategy information data stored in the target network node, determining the strategy characteristics corresponding to the strategy information data, simultaneously acquiring the data structure of the sub-data segment corresponding to the strategy information data, performing fourth matching on the strategy characteristics and the data structure, and evaluating the executable degree of analysis on the corresponding sub-data segment by the strategy information data based on the fourth matching result;
a judging unit configured to:
acquiring an executable degree threshold, comparing the executable degree threshold with the executable degree of the analysis of the corresponding sub-data segment by the strategy information data, and judging whether the strategy information data is qualified or not;
when the executable degree of the strategy information data for analyzing the corresponding sub-data segment is equal to or greater than the executable degree threshold, judging that the strategy information data is qualified, and determining an analysis strategy for analyzing the corresponding sub-data segment based on the strategy information data;
otherwise, judging that the strategy information data is unqualified, and simultaneously, re-matching the target network node in the big data network.
6. The multi-source data oriented big data application development platform of claim 5, wherein the evaluation unit comprises:
the reading subunit is used for reading the data structure, determining the data type, the total data amount and the data logic corresponding to the sub-data segment, simultaneously reading the strategy characteristics, determining the strategy analysis type, the total analysis amount and the strategy analysis logic corresponding to the strategy information data, and performing fourth matching on the data structure and the strategy information data;
a fourth matching unit for:
performing first sub-matching on the data type corresponding to the sub-data segment and the strategy analysis type corresponding to the strategy information data to obtain a first association degree of the sub-data segment and the strategy information data, and determining a first matching weight corresponding to the first sub-matching;
performing second sub-matching on the total data amount corresponding to the sub-data segment and the analysis total amount corresponding to the strategy information data to obtain a second association degree of the sub-data segment and the strategy information data, and determining a second matching weight corresponding to the second sub-matching;
performing third sub-matching on the data logic corresponding to the sub-data segment and the strategy analysis logic corresponding to the strategy information data to obtain a third association degree of the sub-data segment and the strategy information data, and determining a third matching weight corresponding to the third sub-matching;
and the calculating subunit is used for calculating the first association degree, the first matching weight, the second association degree, the second matching weight, the third association degree and the third matching weight, and determining the executable degree of analyzing the corresponding sub-data segment by the strategy information data based on the calculation result.
7. The multi-source data oriented big data application development platform of claim 1, wherein the data analysis module comprises:
the data analysis unit is used for analyzing the multi-source data based on the target decision to obtain an analysis result of analyzing the multi-source data;
and the report generating unit is used for generating an analysis report based on the target decision, the multi-source data and the analysis result.
8. The multi-source data oriented big data application development platform of claim 7, wherein the report generating unit comprises:
the data attribute acquisition subunit is used for acquiring the data attribute of the multi-source data, calling a first file list in a preset document management library based on the data attribute of the multi-source data, automatically inputting the multi-source data into the first file list, and acquiring a first report;
the first copying subunit is used for carrying out first copying on the first report, generating a second file list based on the first copying result adding strategy column, and automatically inputting the target strategy into the second file list to obtain a second report;
and the second copying subunit is used for carrying out second copying on the second report, generating a third file list based on the second copying result adding result column, and automatically inputting the analysis result into the third file list to generate a third report, wherein the third report is an analysis report.
9. The big data application platform for multi-source data according to claim 1, wherein the data analysis module comprises:
an identification unit for determining an analysis result of analyzing the multi-source data based on the analysis report;
the data reading unit is used for reading the multi-source data and determining an analysis result range for analyzing the multi-source data according to the data characteristics of the multi-source data;
a qualification verification unit, configured to:
comparing the analysis result with the analysis result range, and judging whether the analysis report is qualified or not;
when the analysis result is within the analysis result range, judging that the analysis report is qualified;
otherwise, judging that the analysis report is unqualified.
10. The multi-source data oriented big data application platform of claim 1, wherein the storage module comprises:
the storage visual window establishing unit is used for acquiring the data characteristics of the multi-source data and establishing a storage visual window according to the data characteristics of the multi-source data;
the storage visual sub-window establishing unit is used for correspondingly adding the storage visual sub-window according to the visual storage window;
and the visual storage unit is used for storing the multi-source data into the storage visual window, storing the analysis report into the storage visual sub-window, and completing visual storage of the multi-source data and the analysis report uploaded by the client based on the storage visual window and the storage visual sub-window.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310309614.7A CN116483898A (en) | 2023-03-28 | 2023-03-28 | Big data application development platform for multi-source data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310309614.7A CN116483898A (en) | 2023-03-28 | 2023-03-28 | Big data application development platform for multi-source data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116483898A true CN116483898A (en) | 2023-07-25 |
Family
ID=87222268
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310309614.7A Pending CN116483898A (en) | 2023-03-28 | 2023-03-28 | Big data application development platform for multi-source data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116483898A (en) |
-
2023
- 2023-03-28 CN CN202310309614.7A patent/CN116483898A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109510737B (en) | Protocol interface testing method and device, computer equipment and storage medium | |
| CN107357902B (en) | Data table classification system and method based on association rule | |
| CN109783482B (en) | Data violation monitoring method and device | |
| US10521407B2 (en) | Grouping of database objects | |
| CN112685436B (en) | Tracing information processing method and device | |
| KR102474866B1 (en) | Method and apparatus for managing document based on blockchain network | |
| EP4358005A1 (en) | Information recommendation method and apparatus based on data interaction, and device and storage medium | |
| CN110659289B (en) | Material quality detection test report management method and system | |
| CN114297140A (en) | Archive management system based on artificial intelligence | |
| US20240127143A1 (en) | Method, device and storage medium for information processing based on data interaction | |
| CN116483898A (en) | Big data application development platform for multi-source data | |
| WO2022262774A1 (en) | Information exchange method, apparatus and device, and storage medium | |
| CN113610499B (en) | Method and system for job hunting function team occupational credit archives based on blockchain | |
| CN115293871A (en) | Multi-terminal financial data management method and system | |
| CN115438015A (en) | Computer file storage system and method based on block chain | |
| CN114138403A (en) | Mirror image storage and distribution platform | |
| CN113191122A (en) | Service-oriented construction project electronic file and electronic archive four-property detection method | |
| CN112667661B (en) | Tracing information correlation query method and device | |
| CN113949529B (en) | Credible hybrid cloud management platform access method and system | |
| CN117290889B (en) | Safe storage method for realizing electronic labor contract based on blockchain | |
| CN115640369B (en) | Piece information base data storage method applying star-shaped data model | |
| CN114116121A (en) | Container project isolation platform | |
| CN116402477A (en) | File digital information management system | |
| CN107844485A (en) | The update method and device of test script file | |
| CN116932074A (en) | Centralized JAVA code operation configuration method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |