CN116226255A - Efficient batch importing method and system for body measurement data - Google Patents
Efficient batch importing method and system for body measurement data Download PDFInfo
- Publication number
- CN116226255A CN116226255A CN202310247737.2A CN202310247737A CN116226255A CN 116226255 A CN116226255 A CN 116226255A CN 202310247737 A CN202310247737 A CN 202310247737A CN 116226255 A CN116226255 A CN 116226255A
- Authority
- CN
- China
- Prior art keywords
- data
- import
- result
- importing
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 238000005259 measurement Methods 0.000 title claims abstract description 38
- 238000001514 detection method Methods 0.000 claims abstract description 141
- 230000005540 biological transmission Effects 0.000 claims abstract description 132
- 230000005856 abnormality Effects 0.000 claims abstract description 43
- 238000010206 sensitivity analysis Methods 0.000 claims abstract description 11
- 230000008676 import Effects 0.000 claims description 255
- 230000002159 abnormal effect Effects 0.000 claims description 22
- 238000004458 analytical method Methods 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 6
- 238000012545 processing Methods 0.000 abstract description 6
- 230000004069 differentiation Effects 0.000 abstract description 5
- 108091006146 Channels Proteins 0.000 description 91
- 230000008569 process Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 238000012512 characterization method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000012098 association analyses Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000001303 quality assessment method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/26—Government or public services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Human Resources & Organizations (AREA)
- Educational Administration (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Engineering & Computer Science (AREA)
- Development Economics (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Computer And Data Communications (AREA)
- Automatic Analysis And Handling Materials Therefor (AREA)
Abstract
The invention provides a high-efficiency batch importing method and system of body measurement data, which relate to the technical field of data importing and processing, and are used for acquiring a database to be imported, converting the format of an attribute column-data column to generate a batch importing table, carrying out attribute data sensitivity analysis and identifying the sensitive importing table, establishing a data transmission channel group, and importing the sensitive importing table based on a first data transmission channel; the method has the advantages that the residual importing tables are imported through the second data transmission channel, data importing results are obtained, multi-index detection is carried out, and if detection abnormality does not occur, the successful importing results are output, so that the technical problems that in the prior art, the data differentiation is not imported based on a dedicated channel, the final data importing efficiency is low, the importing accuracy is insufficient, and the data is imported in high efficiency and high precision by respectively constructing the transmission channels for dedicated transmission through data standardization and data classification are solved.
Description
Technical Field
The invention relates to the technical field of data importing and processing, in particular to a method and a system for efficiently importing body measurement data in batches.
Background
The school can regularly carry out body measurement to evaluate physical quality of students, and for regular body measurement data, system import is needed to carry out data storage, so that information to be imported is imported into a data management system from a data collection system, and due to the fact that the body measurement data size is large, certain limitations exist on data import efficiency and accuracy in the body measurement data import process, data acquisition can be carried out through an interactive client capable of directly communicating with a data warehouse, data are transmitted to a target data warehouse in a command line or a packet form, and the conventional data import method is still imperfect and needs to be further improved.
In the prior art, the batch data importing method is not intelligent enough, and the data cannot be imported based on a dedicated channel aiming at data differentiation, so that the final data importing efficiency is low, and the importing accuracy is not enough.
Disclosure of Invention
The application provides a high-efficiency batch importing method and system for body measurement data, which are used for solving the technical problems that the batch importing method for data in the prior art is insufficient in intelligence and cannot be used for importing data based on a dedicated channel for data differentiation, so that the importing efficiency of final data is lower and the importing accuracy is insufficient.
In view of the above, the present application provides a method and a system for efficient batch importing of body measurement data.
In a first aspect, the present application provides a method for efficient bulk import of body test data, the method comprising:
connecting the cloud database to obtain a database to be imported;
performing format conversion of attribute column-data column on the database to be imported to generate a batch import table;
performing attribute data sensitivity analysis on the batch import table, and identifying a sensitive import table;
establishing a data transmission channel group based on the importing target of the database to be imported, wherein the data transmission channel group comprises a first data transmission channel and a second data transmission channel;
importing the sensitive importing list through the first data transmission channel to obtain a first data importing result;
importing the rest importing tables except the identification sensitive importing table in the batch importing tables through the second data transmission channel to obtain a second data importing result;
and performing multi-index detection on the first data import result and the second data import result, and outputting a successful import result if no detection abnormality occurs.
In a second aspect, the present application provides an efficient bulk import system of body test data, the system comprising:
the database acquisition module is used for connecting the cloud database to acquire a database to be imported;
the import table generation module is used for carrying out format conversion of attribute columns and data columns on the database to be imported to generate a batch import table;
the sensitive data identification module is used for carrying out attribute data sensitivity analysis on the batch import table and identifying the sensitive import table;
the transmission channel establishment module is used for establishing a data transmission channel group based on the importing target of the database to be imported, wherein the data transmission channel group comprises a first data transmission channel and a second data transmission channel;
the first data import result acquisition module is used for importing the sensitive import table through the first data transmission channel to acquire a first data import result;
the second data import result acquisition module is used for importing the rest import tables except the identification sensitive import table in the batch import tables through the second data transmission channel to acquire a second data import result;
and the import result detection module is used for carrying out multi-index detection on the first data import result and the second data import result, and outputting a successful import result if no detection abnormality occurs.
One or more technical solutions provided in the present application have at least the following technical effects or advantages:
according to the high-efficiency batch importing method of body measurement data, which is provided by the embodiment of the application, the cloud database is connected to obtain a database to be imported, a batch importing table is generated by performing format conversion of attribute columns and data columns, attribute data sensitivity analysis is performed on the batch importing table, and the sensitive importing table is identified; based on the importing target of the database to be imported, establishing a data transmission channel group comprising a first data transmission channel and a second data transmission channel, and importing the sensitive importing list based on the first data transmission channel; the method comprises the steps of importing the rest of the batch importing tables except the identification sensitive importing table through the second data transmission channel, obtaining a first data importing result and a second data importing result, detecting multiple indexes of the first data importing result and the second data importing result, and outputting a successful importing result if no abnormal detection occurs, so that the technical problems that the batch importing method of data in the prior art is insufficient in intelligence and cannot be used for importing the data based on a dedicated channel aiming at data differentiation, the final data importing efficiency is low, and the importing accuracy is insufficient are solved, and the data is imported in high efficiency and high accuracy by respectively constructing the transmission channels for dedicated transmission through data standardization and data classification.
Drawings
FIG. 1 is a schematic flow chart of a method for efficient batch importing of body measurement data;
FIG. 2 is a schematic diagram of a batch import table attribute association flow in a method for efficient batch import of body measurement data;
FIG. 3 is a schematic diagram of a sensitive import table identification flow in a method for efficient bulk import of body measurement data;
fig. 4 is a schematic structural diagram of a system for efficient bulk import of body measurement data.
Reference numerals illustrate: the system comprises a database acquisition module 11, an import table generation module 12, a sensitive data identification module 13, a transmission channel establishment module 14, a first data import result acquisition module 15, a second data import result acquisition module 16 and an import result detection module 17.
Detailed Description
According to the method and the system for efficiently importing the body measurement data in batches, a database to be imported is obtained, format conversion of attribute columns and data columns is carried out to generate a batch importing table, attribute data sensitivity analysis is carried out, sensitive importing tables are identified, a data transmission channel group is established, and the sensitive importing tables are imported based on a first data transmission channel; the method comprises the steps of importing a residual importing list through a second data transmission channel, acquiring a data importing result, performing multi-index detection, and outputting a successful importing result if detection abnormality does not occur, so that the technical problems that in the prior art, the batch importing method of data is insufficient in intelligence and cannot be used for importing data based on a special channel aiming at data differentiation, so that the importing efficiency of final data is low and the importing accuracy is insufficient are solved.
Example 1
As shown in fig. 1, the present application provides a method for efficient batch importing of body measurement data, where the method is applied to a data efficient batch importing system, and the system is communicatively connected to a cloud database, and the method includes:
step S100: connecting the cloud database to obtain a database to be imported;
specifically, a school can regularly perform body measurement to perform physical quality assessment of students, system import is required to perform data storage on regular body measurement data, and due to the fact that the volume of the body measurement data is large, data import efficiency and accuracy in the body measurement data import process are limited to a certain extent.
Step S200: performing format conversion of attribute column-data column on the database to be imported to generate a batch import table;
further, as shown in fig. 2, step S200 of the present application further includes:
step S210: carrying out relevance analysis on the data in the batch import table to obtain attribute relevance coefficients;
step S220: according to the attribute association coefficient, performing column connection on the batch import table to obtain an import table connection result;
step S230: and carrying out connector identification on the connection table based on the connection result of the import table, and realizing the association transmission in the data transmission.
Specifically, the database to be imported is obtained by calling the required data, the database to be imported is subjected to data format conversion, various data attributes are determined, including body measurement items such as 50 meters, 800 meters and long-jump, the integration correspondence of the data attributes and corresponding data is performed, the importing database is converted into a table mode, and the batch importing table is generated.
And further carrying out data relevance analysis on the batch import table, and respectively determining the relevance coefficient between every two data attributes, wherein the similarity of body measurement items and the similarity of data units can be used as a judgment standard. And respectively carrying out index weight configuration on different relevancy decision criteria, namely the body measurement item similarity, the data unit similarity and the like, respectively evaluating relevancy of single criteria, and carrying out weighted calculation to obtain attribute relevancy coefficients. Preferably, the sample data can be determined through big data investigation statistics and similarity measurement is carried out, the judging state is extracted, and a reference judging table is constructed for auxiliary analysis. Further setting a correlation coefficient threshold value, as a coefficient critical value for data attribute correlation judgment, for example, grading based on item correlation, setting a correlation coefficient extremum as 10, setting a level 6 as the correlation coefficient threshold value based on actual judgment requirements, and preferably, for different correlation coefficient levels, performing body measurement data identification based on different identification information so as to improve the characterization definition of the data. When the attribute association coefficient meets the association coefficient threshold, taking the corresponding data attribute as an association attribute; when the data association coefficient does not meet the association attribute threshold, the corresponding data attribute is taken as a non-association attribute, data feature extraction is preferably performed on the body measurement project, whether similar feature attributes exist or not is judged, for example, a data metering mode, a project type and the like, the body measurement data corresponding to 50 meters, 800 meters and 1000 meters are expressed as time, the corresponding data attribute association coefficient is higher and can be regarded as an association attribute, and the body measurement data corresponding to 50 meters and long-jump is not similar or slightly similar, and the corresponding data attribute association coefficient is lower and can be regarded as a non-association attribute.
Further, according to the attribute association coefficient, body measurement data with attribute association in the batch import table is connected, a complete characterization system architecture of the body measurement data is generated, and the complete characterization system architecture is used as a connection result of the import table, namely, data attribute association connection is performed on the batch import table. In order to identify and distinguish different types of associated attributes in the batch import table, a plurality of connectors, namely connection symbols of a plurality of different visual representation modes, are determined, the plurality of connectors correspond to the plurality of associated attributes, the import table connection result is traversed, connection identification is carried out based on the corresponding connectors, and the subsequent targeted transmission processing of data according to the connector identification is facilitated, so that associated transmission in data transmission is realized.
Step S300: performing attribute data sensitivity analysis on the batch import table, and identifying a sensitive import table;
further, as shown in fig. 3, performing attribute data sensitivity analysis on the batch import table, and identifying the sensitive import table, step S300 of the present application further includes:
step S310: acquiring item attribute sets of each table in the batch import table;
step S320: obtaining a preset sensitive information item, and generating a sensitive item traversal library according to the preset sensitive information item;
step S330: performing traversal comparison according to the item attribute set and the sensitive item traversal library, judging whether a sensitive item exists, and acquiring an item identification instruction if the sensitive item exists;
step S340: and identifying an import table of the presence-sensitive item according to the item identification instruction.
Specifically, the batch import table is obtained, the batch import table is provided with associated attribute connectors, item attribute identification and extraction are carried out on each table in the batch import table, the item attribute set is determined, for example, each school, primary school, middle school, high school and university is determined, corresponding body measurement items are different, and the tables of primary school body measurement have parent information and the like; for private information, medical records and the like as personal privacy, the private information, medical records and the like are required to be transmitted through protection of encryption information, related privacy attributes are used as preset sensitive information items, the preset sensitive information items are integrated in a summary mode, the sensitive item traversing library is generated, the item attribute set and the sensitive item traversing library are subjected to traversing comparison, a sensitive item comparison result is obtained, and when data are exported and changed subsequently, if the data relate to sensitive and high-sensitive fields, the DBA and an administrator can set different approval flows to process.
Judging whether a successful matching item exists in the sensitive item matching result, namely, whether a sensitive item exists in the item attribute set, generating the item identification instruction when the successful matching item exists, carrying out reverse matching of the batch import tables based on the sensitive item comparison result along with receiving the item identification instruction, determining the import table of the corresponding sensitive item exists and carrying out identification. The batch import table is divided into the sensitive import table and the common import table by dividing and identifying the batch import table based on whether sensitive items exist or not, so that the batch import table is convenient to carry out targeted transmission subsequently, and specific processing is carried out based on data live.
Step S400: establishing a data transmission channel group based on the importing target of the database to be imported, wherein the data transmission channel group comprises a first data transmission channel and a second data transmission channel;
further, step S400 of the present application further includes:
step S410: setting a batch import mode, wherein the batch import mode is used for configuring activation conditions of the first data transmission channel and the second data transmission channel in the data transmission channel group;
step S420: the batch import mode comprises a very fast import mode and a safe import mode;
step S430: according to the extremely-fast importing mode, configuring transmission parameters for the first data transmission channel and the second data transmission channel, and enabling parallel importing of the same-attribute transmission channels to be achieved;
step S440: and according to the secure import mode, configuring a secure key for the first data transmission channel, and configuring transmission parameters for the second data transmission channel, so as to realize the bipartite import of the transmission channels with different attributes.
Specifically, determining an import target of the database to be imported, for example, setting a data import time limit as the import target, positioning an initial position and an end position of importing execution of data, that is, a current storage position and a final importing storage position, constructing a functional targeted transmission channel by taking the import target, that is, the data import time limit as a limiting factor, specifically, acquiring the constructed transmission channel based on a conventional construction mode, limiting the channel function, and performing mode adjustment and channel activation for different import requirements, for example, data encryption transmission, normal transmission, and the like, to complete data import, wherein the importing execution state of the transmission channel is not fixed, and adjustment and activation can be performed based on the actual import requirement. The data transmission channel group is an auxiliary channel for data import and comprises the first data transmission channel and the second data transmission channel. And further setting the extremely-fast import mode and the safe import mode, and configuring the activation conditions of the first data transmission channel and the second data transmission channel based on the batch import mode as the set batch import mode to realize normal operation of the channels.
Further, when there is no sensitive item in the batch-imported import table, in order to ensure the transmission speed, the extremely-fast import mode needs to be started, and the transmission parameters of the first data transmission channel and the second data transmission channel are configured, including the transmission data amount, the transmission speed, and the like, where the transmission parameters of the two channels are the same, and the parallel import with the same attribute is performed, for example, the 50 meter body measurement item and the 800 meter body measurement item are the same attribute, and the corresponding data can be imported in parallel based on the two channels; when sensitive items exist in the import table imported in batches, namely the sensitive import table, the safe import mode is required to be started, a safety key is configured for the first data transmission channel so as to ensure the data safety in the channel setting process, the data is used for transmitting sensitive item data, a transmission channel is configured for the second data transmission channel so as to transmit non-sensitive item data, at the moment, the two-channel transmission data attribute is different, and the double-split import of the transmission channels with different attributes is realized. The bi-division importing is executed by synchronously importing different data types based on different transmission channels, for example, the encrypted data of the sensitive item data is imported based on the first data transmission channel, the normal importing of the non-sensitive data is executed based on the second data transmission channel, and the data importing is executed by double-channel synchronization. And carrying out importing requirement adjustment on the data transmission channel group based on the attribute of the real-time imported data, so as to realize accurate safety control of data importing and ensure the order of the importing process.
Step S500: importing the sensitive importing list through the first data transmission channel to obtain a first data importing result;
step S600: importing the rest importing tables except the identification sensitive importing table in the batch importing tables through the second data transmission channel to obtain a second data importing result;
specifically, a security key is configured for the first data transmission channel, the first data transmission channel is used as an import channel of the sensitive import table, transmission parameters such as import time limit, import data quantity and the like are determined, the sensitive import table is imported, after data is transmitted to the end of the channel, the storage position of the data is determined, the sensitive import table is stored, and the first data import result is obtained; and determining the rest of the batch import tables except the identification sensitive import table, taking the second data transmission channel as an import channel, determining the import time limit, the import data quantity and other transmission parameters, importing the data, storing the data in corresponding storage positions after the import is completed, generating the second data import result, taking the first data import result and the second data import result as the import result of the batch import table, and further detecting abnormality to ensure the import accuracy.
Step S700: and performing multi-index detection on the first data import result and the second data import result, and outputting a successful import result if no detection abnormality occurs.
Further, the step S700 of the present application further includes:
step S710: constructing a multi-index detection model, wherein the multi-index detection model comprises file type detection, data position detection, writing mode detection and approval result detection;
step S720: respectively inputting the first data input result and the second data input result into the multi-index detection model to obtain a file type detection result, a data position detection result, a writing mode detection result and an approval result detection result;
step S730: judging whether all detection results are in a normal state according to the file type detection result, the data position detection result, the writing mode detection result and the approval result detection result;
step S740: and if all the detection results are in a normal state, outputting a result of successful importing.
Specifically, the sensitive import table and the residual import table in the batch import table are respectively imported in two channels to obtain the first data import result and the second data import result, the multi-index detection model is built to detect states of the first data import result and the second data import result, and the file type, the data position, the writing mode and the approval result are taken as identification nodes and set as a primary identification layer; determining a plurality of sub-nodes corresponding to the identification node, for example, a plurality of refined file types contained in the file type, including a system file, a directory file, a user file and the like, as sub-level branches of the file type, namely a plurality of sub-nodes, setting a secondary identification layer, calling the same type of data information based on big data, namely sample data which is not different from the data type to be imported, carrying out data information identification attribution on the sub-nodes, obtaining a plurality of identification analysis results as decision nodes, carrying out hierarchical corresponding connection on the identification nodes, the matching nodes and the decision nodes, generating a detection decision tree, taking the detection decision tree as construction data, and generating the multi-index detection model through neural network training.
Further, the first data import result and the second data import result are input into the multi-index detection model, and a plurality of decision results are determined and output through multi-level identification matching, wherein the decision results are the file type detection result, the data position detection result, the writing mode detection result and the approval result detection result respectively. Setting a plurality of index detection thresholds, namely, judging whether each detection result is normal or not, and representing whether the detection result is normal or not, wherein the threshold setting can be performed based on professional judgment for judging whether the detection result is normal or not. When all detection results meet the corresponding index detection threshold values, indicating that all detection results are in a normal state, judging that the data is successfully imported, and outputting a successful importing result, namely the importing result of the database to be imported; when any one of the detection results does not meet the corresponding detection index, the abnormal detection result is indicated to exist in the detection results, early warning is carried out on the abnormal detection result so as to carry out adaptive adjustment, abnormal imported data is avoided through imported result detection, and the accuracy of data importing is guaranteed.
Further, step S740 of the present application further includes:
step S741: if all the detection results are in a normal state, performing repeated import detection on the first data import result and the second data import result to obtain repeated import detection results;
step S742: if the repeated import detection result is positive, identifying a repeated import table;
step S743: repeating the import table according to the identification to generate an overlay import table;
step S744: and executing the overlay operation on the identifier repeated import table based on the overlay import table.
Specifically, a plurality of index detection thresholds are set to perform state detection on each detection result, and when each detection result meets the corresponding index detection threshold, that is, each detection result is in a normal state, the first data import result and the second data import result are further subjected to repeated import detection respectively, for example, a table has a multi-import condition, and a repeated import detection result is obtained, including a repeated import table and the number of times of repeated import. And when the repeated import detection result is positive, indicating that a repeated import table exists and identification is carried out, generating the coverage import table, namely the import table needing repeated coverage, wherein multiple groups of the coverage import tables possibly exist, and carrying out repeated coverage operation on the identification repeated import table so as to carry out repeated import table elimination, so that the single import effectiveness of the import table is ensured.
Further, the embodiment of the present application further includes step S750, including:
step S751: if the detection results comprise abnormal detection results, N abnormal detection items are obtained, wherein N is greater than or equal to 1 and less than or equal to 4;
step S752: obtaining N abnormality indexes based on the abnormality duty ratio data of each of the N abnormality detection items;
step S753: performing anomaly level analysis according to the N anomaly indexes to obtain an imported anomaly level;
step S754: and generating abnormal reminding information according to the imported abnormal grade.
Specifically, the detection results are obtained through multi-index detection, when any detection result does not meet the corresponding index detection threshold, the detection result is judged to be in an abnormal state, the abnormality detection result is obtained, N abnormality detection items are obtained, N is greater than or equal to 1 and smaller than or equal to 4, N corresponds to at least one of the file type, the data position, the writing mode and the approval result respectively, one item is immediately extracted based on the N abnormality detection items, for example, for the detection of the file type, whether abnormality exists in a plurality of included refined file types or not is determined, the number of abnormality items is extracted, the ratio calculation is carried out by combining the total file types, the abnormality ratio data in the refined file types is determined, the corresponding abnormality index is obtained, the abnormality index is the characteristic data for detecting the abnormal data quantity, the abnormality index is in proportion to the abnormality ratio data, the N abnormality detection items are respectively analyzed, and the N abnormality indexes are determined. Setting a plurality of level abnormality levels, namely setting a plurality of fixed level standards, carrying out configuration setting by oneself, carrying out abnormality level matching analysis on the N abnormality indexes in proportion to the abnormality indexes, determining the abnormality level to which the N abnormality indexes belong as the imported abnormality level, and generating abnormality reminding information based on the imported abnormality level, wherein the higher the abnormality level is, the higher the intensity of the corresponding abnormality reminding information is, carrying out early warning and warning based on the abnormality reminding information, and adjusting the importing result to ensure the importing accuracy.
Example two
Based on the same inventive concept as the method for efficient bulk import of body test data in the foregoing embodiments, as shown in fig. 4, the present application provides an efficient bulk import system of body test data, the system comprising:
the database acquisition module 11 is used for connecting the cloud database to acquire a database to be imported;
an import table generating module 12, where the import table generating module 12 is configured to perform format conversion of attribute column-data column on the database to be imported to generate a batch import table;
the sensitive data identification module 13 is used for carrying out attribute data sensitivity analysis on the batch import table and identifying the sensitive import table;
a transmission channel establishment module 14, where the transmission channel establishment module 14 is configured to establish a data transmission channel group based on the import target of the database to be imported, where the data transmission channel group includes a first data transmission channel and a second data transmission channel;
a first data import result obtaining module 15, where the first data import result obtaining module 15 is configured to import the sensitive import table through the first data transmission channel to obtain a first data import result;
a second data import result obtaining module 16, where the second data import result obtaining module 16 is configured to import, through the second data transmission channel, the remaining import tables of the batch import tables except the identifier-sensitive import table, so as to obtain a second data import result;
and the lead-in result detection module 17, wherein the lead-in result detection module 17 is used for performing multi-index detection on the first data lead-in result and the second data lead-in result, and outputting a result of successful lead-in if no detection abnormality occurs.
Further, the system further comprises:
the association coefficient acquisition module is used for carrying out association analysis on the data in the batch import table to acquire attribute association coefficients;
the import table connection module is used for carrying out column connection on the batch import tables according to the attribute association coefficients to obtain an import table connection result;
and the connection table identification module is used for carrying out connection identifier identification on the connection table based on the connection result of the lead-in table and is used for realizing the associated transmission in the data transmission.
Further, the system further comprises:
the item attribute acquisition module is used for acquiring item attribute sets of each table in the batch import table;
the traversal library generation module is used for obtaining preset sensitive information items and generating a sensitive item traversal library according to the preset sensitive information items;
the item identification instruction acquisition module is used for performing traversal comparison on the item attribute set and the sensitive item traversal library, judging whether sensitive items exist or not, and acquiring an item identification instruction if the sensitive items exist;
and the sensitive item identification module is used for identifying an import table with sensitive items according to the item identification instruction.
Further, the system further comprises:
the mode setting module is used for setting a batch import mode, wherein the batch import mode is used for configuring the activation conditions of the first data transmission channel and the second data transmission channel in the data transmission channel group;
the system comprises an import mode analysis module, a data processing module and a data processing module, wherein the import mode analysis module is used for the batch import mode comprising a very fast import mode and a safe import mode;
the first transmission parameter configuration module is used for configuring transmission parameters for the first data transmission channel and the second data transmission channel according to the extremely-fast import mode and is used for realizing parallel import of the same-attribute transmission channels;
and the second transmission parameter configuration module is used for configuring a security key for the first data transmission channel according to the security import mode, configuring transmission parameters for the second data transmission channel and realizing the bipartite import of transmission channels with different attributes.
Further, the system further comprises:
the model building module is used for building a multi-index detection model, wherein the multi-index detection model comprises file type detection, data position detection, writing mode detection and approval result detection;
the result detection module is used for respectively inputting the first data import result and the second data import result into the multi-index detection model to obtain a file type detection result, a data position detection result, a writing mode detection result and an approval result detection result;
the result state judging module is used for judging whether all detection results are in a normal state according to the file type detection result, the data position detection result, the writing mode detection result and the approval result detection result;
and the result output module is used for outputting a result of successful import if all the detection results are in a normal state.
Further, the system further comprises:
the detection item acquisition module is used for acquiring N abnormal detection items if the detection results comprise abnormal detection results, wherein N is greater than or equal to 1 and less than or equal to 4;
the abnormality index acquisition module is used for acquiring N abnormality indexes based on the abnormality duty ratio data of each of the N abnormality detection items;
the abnormal grade acquisition module is used for carrying out abnormal grade analysis according to the N abnormal indexes to obtain an imported abnormal grade;
and the information generation module is used for generating abnormal reminding information according to the imported abnormal grade.
Further, the system further comprises:
the repeated import detection module is used for carrying out repeated import detection on the first data import result and the second data import result if all the detection results are in a normal state, so as to obtain repeated import detection results;
the repeated import table identification module is used for identifying the repeated import table if the repeated import detection result is positive;
the coverage import table generation module is used for repeatedly importing a table according to the identification to generate a coverage import table;
and the overlay operation execution module is used for executing overlay operation on the identifier repeated import table based on the overlay import table.
The foregoing detailed description of a method for efficient batch import of body measurement data will be clear to those skilled in the art, and the device disclosed in this embodiment is relatively simple to describe, and the relevant places refer to the method section for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. The utility model provides a high-efficient batch import method of body measurement data, its characterized in that, this method is applied to high-efficient batch import system of data, the system is connected with high in the clouds database communication, this method includes:
connecting the cloud database to obtain a database to be imported;
performing format conversion of attribute column-data column on the database to be imported to generate a batch import table;
performing attribute data sensitivity analysis on the batch import table, and identifying a sensitive import table;
establishing a data transmission channel group based on the importing target of the database to be imported, wherein the data transmission channel group comprises a first data transmission channel and a second data transmission channel;
importing the sensitive importing list through the first data transmission channel to obtain a first data importing result;
importing the rest importing tables except the identification sensitive importing table in the batch importing tables through the second data transmission channel to obtain a second data importing result;
and performing multi-index detection on the first data import result and the second data import result, and outputting a successful import result if no detection abnormality occurs.
2. The method of claim 1, wherein the method further comprises:
carrying out relevance analysis on the data in the batch import table to obtain attribute relevance coefficients;
according to the attribute association coefficient, performing column connection on the batch import table to obtain an import table connection result;
and carrying out connector identification on the connection table based on the connection result of the import table, and realizing the association transmission in the data transmission.
3. The method of claim 1, wherein the batch import table is subjected to attribute data sensitivity analysis, the sensitive import table is identified, the method further comprising:
acquiring item attribute sets of each table in the batch import table;
obtaining a preset sensitive information item, and generating a sensitive item traversal library according to the preset sensitive information item;
performing traversal comparison according to the item attribute set and the sensitive item traversal library, judging whether a sensitive item exists, and acquiring an item identification instruction if the sensitive item exists;
and identifying an import table of the presence-sensitive item according to the item identification instruction.
4. The method of claim 1, wherein the method further comprises:
setting a batch import mode, wherein the batch import mode is used for configuring activation conditions of the first data transmission channel and the second data transmission channel in the data transmission channel group;
the batch import mode comprises a very fast import mode and a safe import mode;
according to the extremely-fast importing mode, configuring transmission parameters for the first data transmission channel and the second data transmission channel, and enabling parallel importing of the same-attribute transmission channels to be achieved;
and according to the secure import mode, configuring a secure key for the first data transmission channel, and configuring transmission parameters for the second data transmission channel, so as to realize the bipartite import of the transmission channels with different attributes.
5. The method of claim 1, wherein the first data import result and the second data import result are multi-index detected, the method further comprising:
constructing a multi-index detection model, wherein the multi-index detection model comprises file type detection, data position detection, writing mode detection and approval result detection;
respectively inputting the first data input result and the second data input result into the multi-index detection model to obtain a file type detection result, a data position detection result, a writing mode detection result and an approval result detection result;
judging whether all detection results are in a normal state according to the file type detection result, the data position detection result, the writing mode detection result and the approval result detection result;
and if all the detection results are in a normal state, outputting a result of successful importing.
6. The method of claim 5, wherein the method further comprises:
if the detection results comprise abnormal detection results, N abnormal detection items are obtained, wherein N is greater than or equal to 1 and less than or equal to 4;
obtaining N abnormality indexes based on the abnormality duty ratio data of each of the N abnormality detection items;
performing anomaly level analysis according to the N anomaly indexes to obtain an imported anomaly level;
and generating abnormal reminding information according to the imported abnormal grade.
7. The method of claim 5, wherein the method further comprises:
if all the detection results are in a normal state, performing repeated import detection on the first data import result and the second data import result to obtain repeated import detection results;
if the repeated import detection result is positive, identifying a repeated import table;
repeating the import table according to the identification to generate an overlay import table;
and executing the overlay operation on the identifier repeated import table based on the overlay import table.
8. An efficient batch import system of body measurement data, wherein the system is in communication connection with a cloud database, the system comprising:
the database acquisition module is used for connecting the cloud database to acquire a database to be imported;
the import table generation module is used for carrying out format conversion of attribute columns and data columns on the database to be imported to generate a batch import table;
the sensitive data identification module is used for carrying out attribute data sensitivity analysis on the batch import table and identifying the sensitive import table;
the transmission channel establishment module is used for establishing a data transmission channel group based on the importing target of the database to be imported, wherein the data transmission channel group comprises a first data transmission channel and a second data transmission channel;
the first data import result acquisition module is used for importing the sensitive import table through the first data transmission channel to acquire a first data import result;
the second data import result acquisition module is used for importing the rest import tables except the identification sensitive import table in the batch import tables through the second data transmission channel to acquire a second data import result;
and the import result detection module is used for carrying out multi-index detection on the first data import result and the second data import result, and outputting a successful import result if no detection abnormality occurs.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310247737.2A CN116226255B (en) | 2023-03-15 | 2023-03-15 | Efficient batch importing method and system for body measurement data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310247737.2A CN116226255B (en) | 2023-03-15 | 2023-03-15 | Efficient batch importing method and system for body measurement data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116226255A true CN116226255A (en) | 2023-06-06 |
| CN116226255B CN116226255B (en) | 2023-08-11 |
Family
ID=86591037
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310247737.2A Active CN116226255B (en) | 2023-03-15 | 2023-03-15 | Efficient batch importing method and system for body measurement data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116226255B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106332070A (en) * | 2015-06-30 | 2017-01-11 | 北京壹人壹本信息科技有限公司 | Secure communication method, device and system |
| CN106503222A (en) * | 2016-11-04 | 2017-03-15 | 上海轻维软件有限公司 | Batch based on Excel imports the method and device of management data base |
| CN112286934A (en) * | 2020-10-29 | 2021-01-29 | 平安信托有限责任公司 | Database table importing method, device, equipment and medium |
| WO2021174696A1 (en) * | 2020-03-06 | 2021-09-10 | 平安科技（深圳）有限公司 | Data updating method and apparatus, computer device, and storage medium |
| WO2022094476A1 (en) * | 2020-11-02 | 2022-05-05 | Sure, Inc. | Method and local and regional cloud infrastructure system for pressure elastography measurement devices |
| CN115617888A (en) * | 2022-09-26 | 2023-01-17 | 中国建设银行股份有限公司 | Data import method, device, equipment, storage medium and product |
-
2023
- 2023-03-15 CN CN202310247737.2A patent/CN116226255B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106332070A (en) * | 2015-06-30 | 2017-01-11 | 北京壹人壹本信息科技有限公司 | Secure communication method, device and system |
| CN106503222A (en) * | 2016-11-04 | 2017-03-15 | 上海轻维软件有限公司 | Batch based on Excel imports the method and device of management data base |
| WO2021174696A1 (en) * | 2020-03-06 | 2021-09-10 | 平安科技（深圳）有限公司 | Data updating method and apparatus, computer device, and storage medium |
| CN112286934A (en) * | 2020-10-29 | 2021-01-29 | 平安信托有限责任公司 | Database table importing method, device, equipment and medium |
| WO2022094476A1 (en) * | 2020-11-02 | 2022-05-05 | Sure, Inc. | Method and local and regional cloud infrastructure system for pressure elastography measurement devices |
| CN115617888A (en) * | 2022-09-26 | 2023-01-17 | 中国建设银行股份有限公司 | Data import method, device, equipment, storage medium and product |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116226255B (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8868985B2 (en) | Supervised fault learning using rule-generated samples for machine condition monitoring | |
| CN110263538A (en) | A kind of malicious code detecting method based on system action sequence | |
| CN109446804B (en) | Intrusion detection method based on multi-scale feature connection convolutional neural network | |
| CN116366374B (en) | Security assessment method, system and medium for power grid network management based on big data | |
| CN111339297A (en) | Network asset anomaly detection method, system, medium, and device | |
| CN110648172B (en) | Identity recognition method and system integrating multiple mobile devices | |
| CN116520068B (en) | Diagnostic method, device, equipment and storage medium for electric power data | |
| CN115222303B (en) | Industry risk data analysis method and system based on big data and storage medium | |
| CN111078512A (en) | Alarm record generation method and device, alarm equipment and storage medium | |
| CN117035563B (en) | Product quality safety risk monitoring method, device, monitoring system and medium | |
| CN116226255B (en) | Efficient batch importing method and system for body measurement data | |
| CN108399387A (en) | The data processing method and device of target group for identification | |
| US20150227639A1 (en) | System data compression system and method thereof | |
| CN114385436A (en) | Server grouping method and device, electronic equipment and storage medium | |
| CN114553790A (en) | Multi-mode feature-based small sample learning Internet of things traffic classification method and system | |
| CN112580781A (en) | Processing method, device and equipment of deep learning model and storage medium | |
| CN109521312B (en) | Non-technical line loss detection method, device and system | |
| CN118245921B (en) | Driving data comparison method and system | |
| CN115473343B (en) | Intelligent gateway multi-master-station parallel access test method | |
| CN112364896B (en) | Method and device for determining health information distribution based on machine learning | |
| CN117318053B (en) | Energy demand prediction method and system for energy storage power station | |
| CN111565187B (en) | DNS (Domain name System) anomaly detection method, device, equipment and storage medium | |
| CN117808558B (en) | Client risk behavior identification method and system based on twin network | |
| CN118193721B (en) | Associated information recommendation system based on natural language processing | |
| Wang | Research on Fault Prevention and Maintenance System of Automatic Substation Primary Equipment Based on Decision Tree Algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |