CN111930723B - Scientific and technological achievement data fusion method based on big data - Google Patents
Scientific and technological achievement data fusion method based on big data Download PDFInfo
- Publication number
- CN111930723B CN111930723B CN202011026936.3A CN202011026936A CN111930723B CN 111930723 B CN111930723 B CN 111930723B CN 202011026936 A CN202011026936 A CN 202011026936A CN 111930723 B CN111930723 B CN 111930723B
- Authority
- CN
- China
- Prior art keywords
- data
- technological achievement
- scientific
- achievement data
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007500 overflow downdraw method Methods 0.000 title claims abstract description 23
- 238000007726 management method Methods 0.000 claims abstract description 49
- 230000000007 visual effect Effects 0.000 claims abstract description 21
- 230000036541 health Effects 0.000 claims abstract description 16
- 230000004927 fusion Effects 0.000 claims abstract description 9
- 238000012544 monitoring process Methods 0.000 claims abstract description 9
- 238000004519 manufacturing process Methods 0.000 claims abstract description 6
- 238000004458 analytical method Methods 0.000 claims abstract description 5
- 238000000034 method Methods 0.000 claims description 33
- 238000000605 extraction Methods 0.000 claims description 27
- 230000005540 biological transmission Effects 0.000 claims description 17
- 238000002347 injection Methods 0.000 claims description 15
- 239000007924 injection Substances 0.000 claims description 15
- 238000012795 verification Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 13
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 230000007246 mechanism Effects 0.000 claims description 12
- 238000003860 storage Methods 0.000 claims description 12
- 238000006243 chemical reaction Methods 0.000 claims description 11
- 230000005856 abnormality Effects 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 8
- 230000006870 function Effects 0.000 claims description 8
- 238000012550 audit Methods 0.000 claims description 7
- 238000013075 data extraction Methods 0.000 claims description 7
- 230000002265 prevention Effects 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 230000000737 periodic effect Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 238000012986 modification Methods 0.000 claims description 5
- 230000004048 modification Effects 0.000 claims description 5
- 238000013524 data verification Methods 0.000 claims description 4
- 230000014509 gene expression Effects 0.000 claims description 4
- 241000700605 Viruses Species 0.000 claims description 3
- 238000013475 authorization Methods 0.000 claims description 3
- 238000004422 calculation algorithm Methods 0.000 claims description 3
- 238000013070 change management Methods 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 235000014510 cooky Nutrition 0.000 claims description 3
- 238000013499 data model Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 230000008676 import Effects 0.000 claims description 3
- 230000009545 invasion Effects 0.000 claims description 3
- 238000011084 recovery Methods 0.000 claims description 3
- 238000012216 screening Methods 0.000 claims description 3
- 230000004083 survival effect Effects 0.000 claims description 3
- 238000007499 fusion processing Methods 0.000 abstract description 6
- 238000011161 development Methods 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000007619 statistical method Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000005111 flow chemistry technique Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/248—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
- G06F16/287—Visualization; Browsing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Abstract
The embodiment of the invention discloses a scientific and technological achievement data fusion method based on big data, which comprises the following steps: constructing a scientific and technological achievement data fusion database based on big data, constructing a scientific and technological achievement data index system, setting health threshold values of various indexes, and monitoring health conditions; visual display is carried out on scientific and technological achievement data by utilizing visual graphics, and automatic analysis and summarization are carried out on application data; the data are subjected to fusion processing, data acquisition is converted, and data of different data platforms are extracted, so that cross-platform, cross-field and cross-professional technological achievement data access is realized; and safety authentication is carried out on the user and the data, so that the risk that technological achievement data relate to secret disclosure and information tampering and malicious codes is prevented. The invention can realize cross-platform, cross-domain and cross-professional technological achievement data access, can conveniently inquire, display and share technological achievement information, deeply excavates technological achievement formed in production and management, and improves management efficiency and business accuracy.
Description
Technical Field
The invention relates to the technical field of data management, in particular to a scientific and technological achievement data fusion method based on big data.
Background
How to realize the deep fusion of scientific and technological achievement data in a cross-platform, cross-field and cross-professional mode, and realize the core of scientific and technological achievement data safety management which can be inquired, counted, analyzed and mined and time data management. The technological achievement data range is wider, the data sources are more, and the technical means of data acquisition and transmission are researched according to the differences of different data sources, so that the problems of non-uniform data formats, non-uniform database types, non-uniform data acquisition modes and the like are solved. Mainly comprises the following steps: firstly, accessing scientific and technological achievement data and industry data of different platforms, different fields and different professions; and secondly, completing multi-source data fusion through statistical analysis and data conversion. In the prior art, the management and mining capability of scientific and technological achievement data is weak, the scientific and technological achievement lacks a unified data format, a data source, a collection method and a use mode, an association relation cannot be formed between the scientific and technological achievement data and project information, statistical analysis is difficult, visual and convenient inquiry of the scientific and technological achievement information cannot be achieved, achievement and process data display is incomplete, data sharing is difficult, development of scientific and technological achievement data fusion and sharing research work is needed, deep mining of achievements formed in production, management and management research is carried out, and management efficiency and business accuracy are improved.
Thus, improvements are needed in the art.
Disclosure of Invention
One technical problem to be solved by the embodiment of the invention is as follows: a scientific and technological achievement data fusion method based on big data is provided to solve the problems existing in the prior art.
The scientific and technological achievement data fusion method based on big data comprises the following steps:
constructing a scientific and technological achievement data fusion database based on big data, constructing a scientific and technological achievement data index system, setting health threshold values of various indexes, and monitoring health conditions;
visual display is carried out on scientific and technological achievement data by utilizing visual graphics, and automatic analysis and summarization are carried out on application data;
the data are subjected to fusion processing, data acquisition is converted, and data of different data platforms are extracted, so that cross-platform, cross-field and cross-professional technological achievement data access is realized;
and safety authentication is carried out on the user and the data, so that the risk that technological achievement data relate to secret disclosure and information tampering and malicious codes is prevented.
In another embodiment of the above-mentioned big data based technological achievement data fusion method of the present invention, the building of the big data based technological achievement data fusion database, the building of the technological achievement data index system, the setting of the health threshold of each index, and the monitoring of the health condition include:
modeling production operation management data and constructing a scientific and technological achievement data index system;
setting a filling template of each index data in a scientific and technological achievement data index system, and receiving and storing the scientific and technological achievement data in real time;
setting specific health thresholds of various indexes and setting early warning conditions of unhealthy indexes;
and monitoring scientific and technological achievement data in the database, and starting an early warning mechanism when the data trigger early warning conditions of unhealthy indexes.
In another embodiment of the above-mentioned big data-based technological achievement data fusion method, the visual display of technological achievement data by using visual graphics includes:
analyzing scientific and technological achievement data through the same ratio, the ring ratio and the duty ratio;
visual display is carried out on scientific and technological achievement data by using a pie chart, a bar chart and a line chart;
a report generation template of the scientific and technological achievement data is formulated, and a scientific and technological achievement data report is automatically generated;
and using a JAVA visual scene to visually display the scientific and technological achievement data model.
In another embodiment of the above-mentioned big data-based technological achievement data fusion method, the data fusion processing, the data acquisition conversion and the data extraction of different data platforms, the implementation of cross-platform, cross-domain and cross-professional technological achievement data access includes:
ETL program development is carried out by applying an information tool, and scientific and technological result data are extracted to a full-service data center;
defining the specification of a scientific and technological achievement data access interface, and defining the time, format and content requirements of data exchange;
the verification of the data interface is completed in a report and log mode, and the extraction of the scientific and technological achievement data comprises initial extraction and periodic extraction, wherein the initial extraction is one-time extraction of related scientific and technological achievement data from a source system, and the periodic extraction is extraction of the data in a designated time according to different data contents and data extraction periods;
judging whether the scientific and technological achievement data are all-service center data, if so, directly importing the scientific and technological achievement data into a database for use;
otherwise, applying for data access, judging whether the scientific and technological achievement data of the application data access support page import;
if yes, importing the scientific and technological achievement data into a database through a system importing interface;
otherwise, the data is input into the database manually.
In another embodiment of the above-mentioned big data-based technological achievement data fusion method of the present invention, the security authentication of the user and the data, and the prevention of technological achievement data from being related to disclosure and falsification of information and malicious code risks include:
authenticating application security of a scientific and technological achievement data user, authenticating user identity information, authorizing user operation, verifying input and output of data, and managing configuration, session, parameter operation and abnormal conditions of a system;
the technical result data is managed safely, so that the storage, transmission, backup and access control of the technical result data are managed safely;
managing the security of an operating system, and carrying out identity authentication, access control restriction, virus invasion prevention, vulnerability scanning, patch updating, security audit, data backup and malicious code prevention on the operating system;
and managing the database security, and carrying out identity authentication, access control, vulnerability scanning, security audit and patch updating on the database system.
In another embodiment of the above big data based scientific and technological achievement data fusion method of the present invention, the authenticating the user identity information includes:
storing the password in a database in an MD5 mode;
the login password is not stored in the cookie, and when the browser is closed, all authentication information is destroyed;
performing password intensity configuration, supporting a password intensity configuration switch and a forced password modification switch, checking password intensity during login, and forcedly modifying the password when the intensity is insufficient;
the login failure times are locked excessively, when the login failure times exceed a set threshold value, an account is automatically locked, and an event is written into a log;
and configuring the IP addresses of the login accounts, and refusing the unified account to login at a plurality of IP addresses.
In another embodiment of the above big data based scientific and technological achievement data fusion method of the present invention, the verifying the input and output of the data includes:
filtering request parameters of the URL, illegal characters of javascript contained in GET and POST data of the form through a filter, and preventing information tampering;
performing input verification based on the regular expression, and meeting the requirements of scientific and technological achievement data of different scenes;
verifying whether the parameters of the sql injection exist in the processes of user login and the like, and preventing the sql injection attack;
based on the uploading file after identity authentication, supporting configuration of the type of the uploading file;
verifying the validity of the attributes of the length, the range and the data type of the data input or imported by the user at the client, prohibiting the input module of the illegal data, and prompting error information;
unifying input and output data verification interfaces, and guaranteeing consistency of verification logic;
formatting the input data according to the data formats of the interfaces, including URL, date, number and character string formats, so as to ensure the data format to be correct;
detecting special characters to prevent XML injection, SQL injection and script injection attack, wherein the special characters comprise single quotation marks, 1=1 and CDATA, for, loop;
and performing data filtering and screening, discarding illegal data, alarming and unifying error prompt information of all integrated interfaces.
In another embodiment of the above big data based technological achievement data fusion method of the present invention, the managing the configuration, the session, the parameter operation and the abnormal situation of the system includes:
system configuration management:
setting a configuration change management flow, and when the configuration change occurs in key information, implementing an approval system, wherein the key information comprises: database connection, operation parameters and template information;
setting a storage and version control rule of configuration information, and avoiding version loss or confusion;
controlling a configuration management role by using a role-based authorization policy, controlling the granularity of configuration authorities, and enabling a configuration management function to be accessed by authorized operators and administrators;
the configuration of database connection, system operation parameters and the like is transmitted to middleware for unified management, and template configuration information is encrypted and stored;
system session management:
setting session survival time, automatically destroying user session after overtime, and deleting session information;
after the user logs in successfully, a new session is established, a session ID is randomly allocated, and the current IP address and the machine name information are bound;
storing session information at a server, and carrying out encryption transmission on user login information and identity credentials;
logging out the login user, and automatically clearing the user session by the server after the user logs out or closes the browser;
system operation parameter management:
verifying the input parameters, and avoiding using query character string parameters containing sensitive data or affecting the security logic of the server;
identifying the client using the session identifier and storing the sensitive item in a session storage area on the server;
HTTP POST is used for replacing GET to submit the window, so that hidden window is avoided;
URL checking is executed at the server side, so that the user is prevented from bypassing the checking without passing through the operation parameters, and the user is prevented from operating the URL parameters through the browser address text box;
limiting the fields that can accept user input and modifying and validating all values from the client;
system abnormal condition management:
uniformly returning a general error message;
using a structured exception handling mechanism and capturing exceptions;
setting an abnormal transaction rollback and compensation mechanism, providing transaction security rollback when an abnormality occurs, and ending the session in which the abnormality occurs.
In another embodiment of the above-mentioned big-data-based technological achievement data fusion method of the present invention, the managing the technological achievement data security to enable the storage, transmission, backup and access control of the data to be managed safely includes:
and (3) safety management of scientific and technological achievement data storage:
when the core management data service information is stored and applied, verifying the integrity of the data, and providing log record tracking and recovering functions aiming at the conditions of data loss and abnormality;
the data deletion is controlled by access, and an access control mechanism of application software is used;
the deletion of data is at least confirmed twice;
the method comprises the steps of performing unidirectional conversion by using an MD5 hash algorithm, and storing the unidirectional conversion in a database, wherein the ciphertext length is 32 bits;
and (3) safety management of scientific and technological achievement data transmission:
transmitting service information by using an HTTPS security protocol, and performing confidentiality protection on remote file access by using SFTP;
encrypting data using a digital signature;
backup safety management of scientific and technological achievement data:
the dual-machine RAC redundant deployment topology is implemented, and service node switching and fault node recovery are carried out when a key node fails;
hardware redundancy of the primary network device, communication lines, and data processing system;
and (3) managing scientific and technological achievement data access control:
setting a sensitive mark for important data resources;
and controlling the operation of the user on the information resource with the sensitive mark according to the security policy.
In another embodiment of the above big data based technological achievement data fusion method of the present invention, the technological achievement data includes structured data and unstructured data;
the access method of the structured data comprises the following steps:
extracting by adopting a ETL, OGG, load mode;
longitudinally interacting by adopting a data transmission assembly;
cleaning and converting by adopting an ETL mode;
adopting an ETL mode to convert and load;
the method for accessing unstructured data comprises the following steps:
collecting by adopting an unstructured data access assembly;
converting by adopting an unstructured conversion component;
and converting and loading by adopting an ETL mode.
Compared with the prior art, the invention has the following advantages:
according to the scientific and technological achievement data fusion method based on big data, a scientific and technological achievement index system is built, health thresholds of indexes are set and monitored, fusion processing is conducted on data, cross-platform, cross-domain and cross-professional scientific and technological achievement data access is met, visual display is conducted on the scientific and technological achievement data, safety protection is conducted on users and the data, comprehensive acquisition of the scientific and technological achievement data is achieved, data are organically fused, and management efficiency and business accuracy are improved.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description, serve to explain the principles of the invention.
The invention may be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:
FIG. 1 is a flow chart of an embodiment of a big data based technological process data fusion method of the present invention.
Detailed Description
Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that: the relative arrangement of the components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless it is specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective parts shown in the drawings are not drawn in actual scale for convenience of description.
The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the invention, its application, or uses.
Techniques, methods, and apparatus known to one of ordinary skill in the relevant art may not be discussed in detail, but are intended to be part of the specification where appropriate.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further discussion thereof is necessary in subsequent figures.
Fig. 1 is a flowchart of an embodiment of a big data based technological process data fusion method according to the present invention, as shown in fig. 1, where the big data based technological process data fusion method includes:
10, constructing a scientific and technological achievement data fusion database based on big data, constructing a scientific and technological achievement data index system, setting health threshold values of various indexes, and monitoring health conditions;
20, carrying out visual display on the scientific and technological achievement data by utilizing a visual graph, and carrying out automatic analysis and summarization on application data;
30, carrying out fusion processing on the data, converting the data acquisition and extracting the data of different data platforms, and realizing cross-platform, cross-field and cross-professional technological achievement data access;
and 40, carrying out security authentication on the user and the data, and preventing the risk of divulging and falsifying information and malicious codes in the scientific and technological achievement data.
Constructing a scientific and technological achievement data fusion database based on big data, constructing a scientific and technological achievement data index system, setting health thresholds of various indexes, and monitoring health conditions comprises the following steps:
modeling production operation management data and constructing a scientific and technological achievement data index system;
setting a filling template of each index data in a scientific and technological achievement data index system, and receiving and storing the scientific and technological achievement data in real time;
setting specific health thresholds of various indexes and setting early warning conditions of unhealthy indexes;
and monitoring scientific and technological achievement data in the database, and starting an early warning mechanism when the data trigger early warning conditions of unhealthy indexes.
In a specific embodiment, the process of modeling production management data includes: the technological achievement data user applies for data access requirements, provides corresponding access schemes for each access data according to structured data and unstructured data, develops a data interface for the selected data access scheme, develops ETL programs by adopting an information tool for data interface development, and extracts source system data to a full-service data center.
In the development process of the data interface, the following judgment needs to be performed:
judging the size of the data volume, and dividing the data interface into incremental access and full access according to the size of the data volume;
the incremental access, the data in the scientific and technological achievement data table with larger data volume, the timestamp, log comparison and full table comparison technology are used, the data of source system change brought by business processing is captured by matching with the change data capturing function provided by the ETL tool, and only the incremental data is transmitted to the data cache area for subsequent ETL flow processing;
full access, which is used for processing objects of a data cache area, wherein the full access is smaller in data volume, or the incremental access technical condition is not provided, or the performance and complexity problems are beyond the tolerance of a service system, and all data in a certain time range are adopted;
the initial data loading adopts a full mode;
interface specification:
in order to ensure accurate and efficient realization of data exchange with a source system, a mode of a data access interface specification is definitely defined, and responsibility of each party, and time, format and content requirements of the data exchange are definitely defined;
the interface specification shall include both interface responsibilities, interface implementations, data interface requirements, and definitions for each detailed interface;
a source data provider for ensuring the on-time provision of data content specified by the interface specification; providing a relevant verification function for the interface data, and ensuring the accuracy and consistency of the data; if the source system is changed, the source system is required to provide a change content description before the change, and the interface specification and the corresponding content of the related accessories are updated in time; following the validation rules specified in the interface specification; the problems occurring in the interface data providing process should be solved in time;
a data receiver receives related data content specified by interface specifications on time; if the interface is changed, providing related description of the changed content before the change, and updating corresponding content of the interface specification and related accessories in time; providing verification of the received data;
the interface implementation, access the source system database through the infromatics database access interface protocol, inquire about the designated interface table to obtain the data to the full-service data center, including: supporting the transmission of an Oracle database access interface, realizing the database connection and data acquisition of a local area network and a wide area network platform, and supporting the database reconnection and data retransmission; the verification of IP address, account and password of the interface machine is supported;
extraction period of interface data:
the method comprises the steps of initial extraction, namely extracting relevant data from a source system at one time, wherein the data range of the initial extraction comprises all data with valid latest state during extraction, and the data period during the initial extraction is described in the description of a data interface unit according to specific conditions;
the method comprises the steps of regularly extracting, namely, according to different data contents and data extraction periods, carrying out data extraction in a specified time, wherein the regular extraction is divided into a total daily amount, a daily increment and a total monthly amount;
the interface verification mode is as follows:
verifying the data interface, wherein the verification of the interface data is jointly responsible for both sides of the interface, and the verification is performed in a report form checking and log mode;
the report forms are subjected to verification of data consistency through the audit between the report forms and indexes with correlation relations;
and the log is verified by comparing the processing logs of the data receiver and the data provider.
The data interface defines in detail:
for each data interface that needs to be transmitted, the following is defined:
data content, interface unit name, code, description, interface unit corresponding list main key;
interface unit field list, field name, service description, field type, field format, field length, whether null value is allowed;
extracting mode and period;
OGG synchronization process configuration:
and carrying out data synchronization by adopting an OGG mode, carrying out OGG process configuration on a source database and an SG-RDB end, and adding synchronization table information into a process to realize real-time data synchronization.
Visual display is carried out on scientific and technological achievement data by utilizing visual graphics, and automatic analysis and summarization of application data comprise the following steps:
analyzing scientific and technological achievement data through the same ratio, the ring ratio and the duty ratio;
visual display is carried out on scientific and technological achievement data by using a pie chart, a bar chart and a line chart;
a report generation template of the scientific and technological achievement data is formulated, and a scientific and technological achievement data report is automatically generated;
and using a JAVA visual scene to visually display the scientific and technological achievement data model.
The data fusion processing is performed, the data acquisition is converted and the data extraction of different data platforms is performed, and the cross-platform, cross-domain and cross-professional technological achievement data access is realized, which comprises the following steps:
ETL program development is carried out by applying an information tool, and scientific and technological result data are extracted to a full-service data center;
defining the specification of a scientific and technological achievement data access interface, and defining the time, format and content requirements of data exchange;
the verification of the data interface is completed in a report and log mode, and the extraction of the scientific and technological achievement data comprises initial extraction and periodic extraction, wherein the initial extraction is one-time extraction of related scientific and technological achievement data from a source system, and the periodic extraction is extraction of the data in a designated time according to different data contents and data extraction periods;
judging whether the scientific and technological achievement data are all-service center data, if so, directly importing the scientific and technological achievement data into a database for use;
otherwise, applying for data access, judging whether the scientific and technological achievement data of the application data access support page import;
if yes, importing the scientific and technological achievement data into a database through a system importing interface;
otherwise, the data is input into the database manually.
The security authentication of the user and the data, preventing technological achievement data from being related to secret disclosure and information falsification and malicious code risks, comprises:
authenticating application security of a scientific and technological achievement data user, authenticating user identity information, authorizing user operation, verifying input and output of data, and managing configuration, session, parameter operation and abnormal conditions of a system;
the technical result data is managed safely, so that the storage, transmission, backup and access control of the technical result data are managed safely;
managing the security of an operating system, and carrying out identity authentication, access control restriction, virus invasion prevention, vulnerability scanning, patch updating, security audit, data backup and malicious code prevention on the operating system;
and managing the database security, and carrying out identity authentication, access control, vulnerability scanning, security audit and patch updating on the database system.
The authentication of the user identity information comprises the following steps:
storing the password in a database in an MD5 mode;
the login password is not stored in the cookie, and when the browser is closed, all authentication information is destroyed;
performing password intensity configuration, supporting a password intensity configuration switch and a forced password modification switch, checking password intensity during login, and forcedly modifying the password when the intensity is insufficient;
the login failure times are locked excessively, when the login failure times exceed a set threshold value, an account is automatically locked, and an event is written into a log;
and configuring the IP addresses of the login accounts, and refusing the unified account to login at a plurality of IP addresses.
The verifying the input and output of the data comprises the following steps:
filtering request parameters of the URL, illegal characters of javascript contained in GET and POST data of the form through a filter, and preventing information tampering;
performing input verification based on the regular expression, and meeting the requirements of scientific and technological achievement data of different scenes;
verifying whether the parameters of the sql injection exist in the processes of user login and the like, and preventing the sql injection attack;
based on the uploading file after identity authentication, supporting configuration of the type of the uploading file;
verifying the validity of the attributes of the length, the range and the data type of the data input or imported by the user at the client, prohibiting the input module of the illegal data, and prompting error information;
unifying input and output data verification interfaces, and guaranteeing consistency of verification logic;
formatting the input data according to the data formats of the interfaces, including URL, date, number and character string formats, so as to ensure the data format to be correct;
detecting special characters to prevent XML injection, SQL injection and script injection attack, wherein the special characters comprise single quotation marks, 1=1 and CDATA, for, loop;
and performing data filtering and screening, discarding illegal data, alarming and unifying error prompt information of all integrated interfaces.
The management of configuration, session, parameter operation and abnormal conditions of the system comprises:
system configuration management:
setting a configuration change management flow, and when the configuration change occurs in key information, implementing an approval system, wherein the key information comprises: database connection, operation parameters and template information;
setting a storage and version control rule of configuration information, and avoiding version loss or confusion;
controlling a configuration management role by using a role-based authorization policy, controlling the granularity of configuration authorities, and enabling a configuration management function to be accessed by authorized operators and administrators;
the configuration of database connection, system operation parameters and the like is transmitted to middleware for unified management, and template configuration information is encrypted and stored;
system session management:
setting session survival time, automatically destroying user session after overtime, and deleting session information;
after the user logs in successfully, a new session is established, a session ID is randomly allocated, and the current IP address and the machine name information are bound;
storing session information at a server, and carrying out encryption transmission on user login information and identity credentials;
logging out the login user, and automatically clearing the user session by the server after the user logs out or closes the browser;
system operation parameter management:
verifying the input parameters, and avoiding using query character string parameters containing sensitive data or affecting the security logic of the server;
identifying the client using the session identifier and storing the sensitive item in a session storage area on the server;
HTTP POST is used for replacing GET to submit the window, so that hidden window is avoided;
URL checking is executed at the server side, so that the user is prevented from bypassing the checking without passing through the operation parameters, and the user is prevented from operating the URL parameters through the browser address text box;
limiting the fields that can accept user input and modifying and validating all values from the client;
system abnormal condition management:
uniformly returning a general error message;
using a structured exception handling mechanism and capturing exceptions;
setting an abnormal transaction rollback and compensation mechanism, providing transaction security rollback when an abnormality occurs, and ending the session in which the abnormality occurs.
The managing the safety of the technological achievement data to ensure that the safety management of the storage, transmission, backup and access control of the data comprises the following steps:
and (3) safety management of scientific and technological achievement data storage:
when the core management data service information is stored and applied, verifying the integrity of the data, and providing log record tracking and recovering functions aiming at the conditions of data loss and abnormality;
the data deletion is controlled by access, and an access control mechanism of application software is used;
the deletion of data is at least confirmed twice;
the method comprises the steps of performing unidirectional conversion by using an MD5 hash algorithm, and storing the unidirectional conversion in a database, wherein the ciphertext length is 32 bits;
and (3) safety management of scientific and technological achievement data transmission:
transmitting service information by using an HTTPS security protocol, and performing confidentiality protection on remote file access by using SFTP;
encrypting data using a digital signature;
backup safety management of scientific and technological achievement data:
the dual-machine RAC redundant deployment topology is implemented, and service node switching and fault node recovery are carried out when a key node fails;
hardware redundancy of the primary network device, communication lines, and data processing system;
and (3) managing scientific and technological achievement data access control:
setting a sensitive mark for important data resources;
and controlling the operation of the user on the information resource with the sensitive mark according to the security policy.
The scientific and technological achievement data comprises structured data and unstructured data;
the access method of the structured data comprises the following steps:
extracting by adopting a ETL, OGG, load mode;
longitudinally interacting by adopting a data transmission assembly;
cleaning and converting by adopting an ETL mode;
adopting an ETL mode to convert and load;
the method for accessing unstructured data comprises the following steps:
collecting by adopting an unstructured data access assembly;
converting by adopting an unstructured conversion component;
and converting and loading by adopting an ETL mode.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different manner from other embodiments, so that the same or similar parts between the embodiments are mutually referred to. For system embodiments, the description is relatively simple as it essentially corresponds to method embodiments, and reference should be made to the description of method embodiments for relevant points.
The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
Claims (4)
1. A technological achievement data fusion method based on big data is characterized by comprising the following steps:
constructing a scientific and technological achievement data fusion database based on big data, constructing a scientific and technological achievement data index system, setting health threshold values of various indexes, and monitoring health conditions, wherein the method comprises the following steps:
modeling production operation management data and constructing a scientific and technological achievement data index system;
setting a filling template of each index data in a scientific and technological achievement data index system, and receiving and storing the scientific and technological achievement data in real time;
setting specific health thresholds of various indexes and setting early warning conditions of unhealthy indexes;
monitoring scientific and technological achievement data in a database, and starting an early warning mechanism when the data trigger early warning conditions of unhealthy indexes;
visual display is carried out on scientific and technological achievement data by utilizing visual graphics, automatic analysis and summarization are carried out on application data, and the method comprises the following steps:
analyzing scientific and technological achievement data through the same ratio, the ring ratio and the duty ratio;
visual display is carried out on scientific and technological achievement data by using a pie chart, a bar chart and a line chart;
a report generation template of the scientific and technological achievement data is formulated, and a scientific and technological achievement data report is automatically generated;
using JAVA visual scene to perform visual display on the scientific and technological achievement data model;
the data are fused, the data acquisition is converted and the data of different data platforms are extracted, so that cross-platform, cross-field and cross-professional technological achievement data access is realized, and the method comprises the following steps:
ETL program development is carried out by applying an information tool, and scientific and technological result data are extracted to a full-service data center;
defining the specification of a scientific and technological achievement data access interface, and defining the time, format and content requirements of data exchange;
the verification of the data interface is completed in a report and log mode, and the extraction of the scientific and technological achievement data comprises initial extraction and periodic extraction, wherein the initial extraction is one-time extraction of related scientific and technological achievement data from a source system, and the periodic extraction is extraction of the data in a designated time according to different data contents and data extraction periods;
judging whether the scientific and technological achievement data are all-service center data, if so, directly importing the scientific and technological achievement data into a database for use;
otherwise, applying for data access, judging whether the scientific and technological achievement data of the application data access support page import;
if yes, importing the scientific and technological achievement data into a database through a system importing interface;
otherwise, the data is input into the database in a manual mode;
safety authentication is carried out on users and data, and technological achievement data is prevented from being related to secret leakage and information tampering and malicious code risks, and the safety authentication method comprises the following steps:
authenticating application security of a scientific and technological achievement data user, authenticating user identity information, authorizing user operation, verifying input and output of data, and managing configuration, session, parameter operation and abnormal conditions of a system;
the technical result data is managed safely, so that the storage, transmission, backup and access control of the technical result data are managed safely;
managing the security of an operating system, and carrying out identity authentication, access control restriction, virus invasion prevention, vulnerability scanning, patch updating, security audit, data backup and malicious code prevention on the operating system;
managing database security, and carrying out identity authentication, access control, vulnerability scanning, security audit and patch updating on a database system;
the management of configuration, session, parameter operation and abnormal conditions of the system comprises:
system configuration management:
setting a configuration change management flow, and when the configuration change occurs in key information, implementing an approval system, wherein the key information comprises: database connection, operation parameters and template information;
setting a storage and version control rule of configuration information, and avoiding version loss or confusion;
controlling a configuration management role by using a role-based authorization policy, controlling the granularity of configuration authorities, and enabling a configuration management function to be accessed by authorized operators and administrators;
the database connection and the system operation parameter configuration are communicated with middleware for unified management, and the template configuration information is encrypted and stored;
system session management:
setting session survival time, automatically destroying user session after overtime, and deleting session information;
after the user logs in successfully, a new session is established, a session ID is randomly allocated, and the current IP address and the machine name information are bound;
storing session information at a server, and carrying out encryption transmission on user login information and identity credentials;
logging out the login user, and automatically clearing the user session by the server after the user logs out or closes the browser;
system operation parameter management:
verifying the input parameters, and avoiding using query character string parameters containing sensitive data or affecting the security logic of the server;
identifying the client using the session identifier and storing the sensitive item in a session storage area on the server;
HTTP POST is used for replacing GET to submit the window, so that hidden window is avoided;
URL checking is executed at the server side, so that the user is prevented from bypassing the checking without passing through the operation parameters, and the user is prevented from operating the URL parameters through the browser address text box;
limiting the fields that can accept user input and modifying and validating all values from the client;
system abnormal condition management:
uniformly returning a general error message;
using a structured exception handling mechanism and capturing exceptions;
setting an abnormal transaction rollback and compensation mechanism, providing transaction security rollback when an abnormality occurs, and ending the session in which the abnormality occurs;
the managing the safety of the technological achievement data to ensure that the safety management of the storage, transmission, backup and access control of the data comprises the following steps:
and (3) safety management of scientific and technological achievement data storage:
when the core management data service information is stored and applied, verifying the integrity of the data, and providing log record tracking and recovering functions aiming at the conditions of data loss and abnormality;
the data deletion is controlled by access, and an access control mechanism of application software is used;
the deletion of data is at least confirmed twice;
the method comprises the steps of performing unidirectional conversion by using an MD5 hash algorithm, and storing the unidirectional conversion in a database, wherein the ciphertext length is 32 bits;
and (3) safety management of scientific and technological achievement data transmission:
transmitting service information by using an HTTPS security protocol, and performing confidentiality protection on remote file access by using SFTP;
encrypting data using a digital signature;
backup safety management of scientific and technological achievement data:
the dual-machine RAC redundant deployment topology is implemented, and service node switching and fault node recovery are carried out when a key node fails;
hardware redundancy of the primary network device, communication lines, and data processing system;
and (3) managing scientific and technological achievement data access control:
setting a sensitive mark for important data resources;
and controlling the operation of the user on the information resource with the sensitive mark according to the security policy.
2. The big data based technological achievement data fusion method of claim 1, wherein the authentication of the user identity information includes:
storing the password in a database in an MD5 mode;
the login password is not stored in the cookie, and when the browser is closed, all authentication information is destroyed;
performing password intensity configuration, supporting a password intensity configuration switch and a forced password modification switch, checking password intensity during login, and forcedly modifying the password when the intensity is insufficient;
the login failure times are locked excessively, when the login failure times exceed a set threshold value, an account is automatically locked, and an event is written into a log;
and configuring the IP addresses of the login accounts, and refusing the unified account to login at a plurality of IP addresses.
3. The big data based technological achievement data fusion method of claim 1, wherein the verifying the input and output of the data comprises:
filtering request parameters of the URL, illegal characters of javascript contained in GET and POST data of the form through a filter, and preventing information tampering;
performing input verification based on the regular expression, and meeting the requirements of scientific and technological achievement data of different scenes;
verifying whether the user login process has the parameters of sql injection or not, and preventing the sql injection attack;
based on the uploading file after identity authentication, supporting configuration of the type of the uploading file;
verifying the validity of the attributes of the length, the range and the data type of the data input or imported by the user at the client, prohibiting the input module of the illegal data, and prompting error information;
unifying input and output data verification interfaces, and guaranteeing consistency of verification logic;
formatting the input data according to the data formats of the interfaces, including URL, date, number and character string formats, so as to ensure the data format to be correct;
detecting special characters to prevent XML injection, SQL injection and script injection attack, wherein the special characters comprise single quotation marks, 1=1 and CDATA, for, loop;
and performing data filtering and screening, discarding illegal data, alarming and unifying error prompt information of all integrated interfaces.
4. The big data based technological achievement data fusion method of claim 1, wherein the technological achievement data comprises structured data and unstructured data;
the access method of the structured data comprises the following steps:
extracting by adopting a ETL, OGG, load mode;
longitudinally interacting by adopting a data transmission assembly;
cleaning and converting by adopting an ETL mode;
adopting an ETL mode to convert and load;
the method for accessing unstructured data comprises the following steps:
collecting by adopting an unstructured data access assembly;
converting by adopting an unstructured conversion component;
and converting and loading by adopting an ETL mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011026936.3A CN111930723B (en) | 2020-09-25 | 2020-09-25 | Scientific and technological achievement data fusion method based on big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011026936.3A CN111930723B (en) | 2020-09-25 | 2020-09-25 | Scientific and technological achievement data fusion method based on big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111930723A CN111930723A (en) | 2020-11-13 |
| CN111930723B true CN111930723B (en) | 2024-02-02 |
Family
ID=73334274
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011026936.3A Active CN111930723B (en) | 2020-09-25 | 2020-09-25 | Scientific and technological achievement data fusion method based on big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111930723B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112965858B (en) * | 2021-03-04 | 2023-07-18 | 电信科学技术第五研究所有限公司 | Method for realizing networking distributed storage data conflict processing |
| CN113111105A (en) * | 2021-04-06 | 2021-07-13 | 创意信息技术股份有限公司 | Data customized access method and system based on big data |
| CN113723747A (en) * | 2021-07-16 | 2021-11-30 | 北京房江湖科技有限公司 | Analysis report generation method, electronic device and readable storage medium |
| CN115987579B (en) * | 2022-12-07 | 2023-09-15 | 南京鼎山信息科技有限公司 | Data processing method and data processing system based on big data and Internet of things communication |
| CN116915500B (en) * | 2023-09-05 | 2023-11-17 | 武汉万数科技有限公司 | Security detection method and system for access equipment |
| CN116910036B (en) * | 2023-09-11 | 2023-12-26 | 中国林业科学研究院森林生态环境与自然保护研究所(国家林业和草原局世界自然遗产保护研究中心) | Construction method of multisource forest soil attribute database |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106354786A (en) * | 2016-08-23 | 2017-01-25 | 冯村 | Visual analysis method and system |
| CN106777227A (en) * | 2016-12-26 | 2017-05-31 | 河南信安通信技术股份有限公司 | Multidimensional data convergence analysis system and method based on cloud platform |
| CN111125228A (en) * | 2019-12-26 | 2020-05-08 | 北京航天泰坦科技股份有限公司 | Data sharing method and device based on forestry data sharing service platform |
| CN111191259A (en) * | 2019-12-24 | 2020-05-22 | 绍兴市上虞区理工高等研究院 | Scientific and technological achievement encryption transmission method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9914054B2 (en) * | 2014-06-07 | 2018-03-13 | Microsoft Technology Licensing, Llc | Display of system-level achievements with real-time updating |
-
2020
- 2020-09-25 CN CN202011026936.3A patent/CN111930723B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106354786A (en) * | 2016-08-23 | 2017-01-25 | 冯村 | Visual analysis method and system |
| CN106777227A (en) * | 2016-12-26 | 2017-05-31 | 河南信安通信技术股份有限公司 | Multidimensional data convergence analysis system and method based on cloud platform |
| CN111191259A (en) * | 2019-12-24 | 2020-05-22 | 绍兴市上虞区理工高等研究院 | Scientific and technological achievement encryption transmission method and device |
| CN111125228A (en) * | 2019-12-26 | 2020-05-08 | 北京航天泰坦科技股份有限公司 | Data sharing method and device based on forestry data sharing service platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111930723A (en) | 2020-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111930723B (en) | Scientific and technological achievement data fusion method based on big data | |
| CN109831327B (en) | IMS full-service network monitoring intelligent operation and maintenance support system based on big data analysis | |
| CN110881044B (en) | Computer firewall dynamic defense security platform | |
| KR101327317B1 (en) | Apparatus and method for sap application traffic analysis and monitoring, and the information protection system thereof | |
| CN112765245A (en) | Electronic government affair big data processing platform | |
| CN112217835B (en) | Message data processing method and device, server and terminal equipment | |
| US8285778B2 (en) | Protecting web application data | |
| EP2144420A1 (en) | Web application security filtering | |
| KR100912794B1 (en) | Web hacking management system and manegement method thereof for real time web server hacking analysis and homepage hacking search | |
| CN102868702B (en) | System login device and system login method | |
| CN109977673B (en) | Vulnerability repairing method and system based on web website system security | |
| CN103413083A (en) | Security defending system for single host | |
| CN103441926A (en) | Security gateway system of numerically-controlled machine tool network | |
| CN106161710B (en) | A kind of user account safety management system based on smart phone | |
| CN112383535B (en) | Method and device for detecting Hash transfer attack behavior and computer equipment | |
| CN115270182A (en) | Power grid project closed-loop control file management system | |
| CN108769004B (en) | Remote operation safety verification method for industrial internet intelligent equipment | |
| CN101408955A (en) | Method and system determining obligation base on tactic | |
| CN111652454A (en) | Supervision quality and safety production management evaluation management system | |
| Mohsin et al. | Uml-sr: A novel security requirements specification language | |
| KR20070109775A (en) | The security method for authentication which using of random password | |
| CN114881469A (en) | Performance assessment and management system and method for enterprise workers | |
| KR102192232B1 (en) | System for providing verification and guide line of cyber security based on block chain | |
| CN107248037A (en) | A kind of long-distance monitoring method for financial data | |
| KR101425720B1 (en) | Method for Managing Database Security |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |