CN115617780A - Data import method, device, equipment and storage medium - Google Patents
Data import method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN115617780A CN115617780A CN202211300021.6A CN202211300021A CN115617780A CN 115617780 A CN115617780 A CN 115617780A CN 202211300021 A CN202211300021 A CN 202211300021A CN 115617780 A CN115617780 A CN 115617780A
- Authority
- CN
- China
- Prior art keywords
- data
- import
- record
- module
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/177—Editing, e.g. inserting or deleting of tables; using ruled lines
- G06F40/18—Editing, e.g. inserting or deleting of tables; using ruled lines of spreadsheets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/186—Templates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data processing, in particular to a data import method, a data import device, data import equipment and a storage medium, wherein the data import method is applied to a data import system, and the data import system comprises the following components: the data importing method comprises the following steps of performing man-machine interaction with service personnel, and comprises an interaction interface, a storage module, a creation module and an importing module, wherein the data importing method comprises the following steps: receiving and storing a table creation rule configured by a business person based on an interactive interface through a storage module; when a template generation instruction is received, calling a target rule corresponding to the template generation instruction from a storage module through a creation module to generate a template file, so that business personnel can use the template file to perform data entry to obtain a data file and upload the data file; receiving a data file uploaded by a service worker based on an interactive interface; and analyzing the data file into a data record through the import module, and storing the data record into a database. The invention realizes the improvement of the convenience of data import.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data importing method, apparatus, device, and computer readable storage medium.
Background
In the supply chain business circulation, there are links which need to be manually performed off-line, such as calling out, passing through meetings, client follow-up and the like. After such a link is completed, the related data generally needs to be imported into the online service system, so that the service system processes the imported data and generates a processing result. The supply chain manager can analyze the most valuable data according to the processing result, discover the deficiency and defect of the supply chain in time and conveniently monitor the state of the supply chain, thereby optimizing and improving the supply chain and improving the overall capacity of the supply chain.
In the existing data import process, data is imported after data entry is carried out by using a fixed form template. At present, creating or modifying a form template generally requires a development designer to redeploy an updated form template for data import after creating a template file in a coding environment. This makes the process of data import complicated, loaded down with trivial details, has influenced the convenience of data import.
Disclosure of Invention
The invention mainly aims to provide a data import method, a data import device, data import equipment and a computer readable storage medium, and aims to improve the convenience of data import.
In order to achieve the above object, the present invention provides a data importing method, which is applied to a data importing system, the data importing system including: the system comprises an interaction interface, a storage module, a creation module and an import module, wherein the interaction interface is used for performing human-computer interaction with service personnel;
the data import method comprises the following steps:
receiving and storing a table creation rule configured by the service personnel based on the interactive interface through the storage module;
when a template generation instruction is received, calling a target rule corresponding to the template generation instruction from the storage module through the creation module to generate a template file, so that the service personnel can use the template file to perform data entry to obtain a data file and upload the data file;
receiving the data file uploaded by the service personnel based on the interactive interface;
and analyzing the data file into a data record through the import module, and storing the data record into a database.
Optionally, before the step of storing the data record in the database, the data importing method further includes:
for any target record in the data records, verifying the target record through the import module;
and if the verification is passed, executing the step of storing the data record into a database.
Optionally, after the step of verifying the target record by the importing module, the data importing method further includes:
if the verification fails, displaying a verification failure record through the interactive interface so that the business personnel can modify and upload the verification failure record;
and receiving the modified verification failure record, taking the verification failure record as the target record, and executing the step of verifying the target record through the import module.
Optionally, the table creation rule includes: the field encoding method comprises the following steps of field encoding, field description and field rules, wherein the field rules comprise rules for representing mandatory filling identification, data types, data sequences and filling formats;
the step of verifying the target record through the import module includes:
detecting whether each data in the target record meets the field rule or not through the import module;
if yes, determining that the target record passes verification;
if not, determining that the target record is not verified.
Optionally, before the step of parsing the data file into data records by the importing module, the data importing method further includes:
detecting whether the data volume of the data file is larger than a preset threshold value or not;
if the data volume of the data file is less than or equal to the preset threshold value, the step of analyzing the data file into a data record through the import module is executed;
if the data volume of the data file is larger than the preset threshold value, the data file is split into multiple subfiles containing the preset data volume through the import module, and each subfile is analyzed into the data record.
Optionally, the data import system further includes a processing module;
the data import method further comprises the following steps:
sending the address information of the data record in the database to a data processing terminal through the processing module, so that the data processing terminal reads the data record from the database based on the address information, performs data processing on the data record to generate a processing result and returns the processing result;
receiving the processing result, and if the processing result is determined to be an abnormal result representing that an abnormal data record exists, acquiring the abnormal data record from the database through the processing module based on the abnormal result;
displaying the abnormal data record through the interactive interface so that the business personnel can modify and upload the abnormal data record;
and receiving the modified abnormal data record, and storing the modified abnormal data record into the database through the import module so that the data processing terminal can process data based on the modified abnormal data record.
Optionally, the step of storing the modified abnormal data record into the database through the import module includes:
reading a data identifier of the abnormal data record through the import module;
replacing, by the import module, the exception data record with the modified exception data record based on the address information and the data identifier to store the modified exception data record in the database.
In addition, to achieve the above object, the present invention further provides a data importing apparatus, where the data importing apparatus is applied to a data importing system, and the data importing system includes: the system comprises an interaction interface, a storage module, a creation module and an import module, wherein the interaction interface is used for performing human-computer interaction with service personnel;
the data import apparatus includes:
the first calling module is used for receiving and storing the table creation rule configured by the service personnel based on the interactive interface through the storage module;
the second calling module is used for calling a target rule corresponding to the template generation instruction from the storage module through the creating module to generate a template file when the template generation instruction is received, so that the service personnel can use the template file to perform data entry to obtain a data file and upload the data file;
the third calling module is used for receiving the data file uploaded by the service personnel based on the interactive interface;
and the fourth calling module is used for analyzing the data file into a data record through the importing module and storing the data record into a database.
In addition, to achieve the above object, the present invention further provides a data importing apparatus, which includes a memory, a processor, and a data importing program stored in the memory and executable on the processor, and when executed by the processor, the data importing program implements the steps of the data importing method.
In addition, to achieve the above object, the present invention also provides a computer readable storage medium having a data import program stored thereon, where the data import program realizes the steps of the above data import method when being executed by a processor.
The method comprises the steps that a storage module receives and stores a form creation rule configured by a service person based on an interactive interface, when a template generation instruction is received, a creation module calls a target rule corresponding to the template generation instruction from the storage module to generate a template file so that the service person can use the template file to perform data entry to obtain and upload a data file, the data file uploaded by the service person based on the interactive interface is received, the data file is analyzed into a data record through an import module, and the data record is stored in a database.
The method and the device are different from the method and the device for importing data after the template file for importing data is created in the coding environment, so that the operation of business personnel is simplified, the convenience for creating the template file for importing data is improved, and the convenience for importing data is improved. In addition, the invention does not need business personnel to carry out complex coding, reduces the time for creating the template file for data import, and improves the working efficiency of data import.
Meanwhile, the invention does not need the service personnel to create the template file in the coding environment, so the invention does not depend on the operation and the technology of the service personnel when creating the template file for importing the data, and improves the operability of importing the data.
Drawings
FIG. 1 is a flowchart illustrating a data importing method according to a first embodiment of the present invention;
fig. 2 is a schematic functional block diagram of a data importing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data importing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
An embodiment of the present invention provides a data importing system, where the data importing system includes: the system comprises an interactive interface for man-machine interaction with business personnel, a storage module, a creation module and an import module. In particular, the amount of the solvent to be used,
in this embodiment, an interactive interface for human-computer interaction with a service person may provide operations for the service person to create a template file form creation rule, download a file, and upload a file. Further, in the specific implementation, the interactive interface may also provide other functions, and the interactive interface may be specifically designed according to the needs of business personnel.
In this embodiment, the table creation rule configured by the service person based on the interactive interface is received and stored by the storage module. In particular embodiments, the table creation rules include: copy
And when a template generation instruction is received, calling a target rule corresponding to the template generation instruction from the storage module through the creation module to generate a template file. After the template file is generated through the creation module, the data file can be obtained by data entry of business personnel through a file downloading function provided by the interactive interface.
Through a file uploading function provided by the interactive interface, business personnel can upload the template file (hereinafter referred to as the data file for distinction) after data is input by using the template file to obtain the data file. After receiving the data file uploaded by the service personnel, the data file can be analyzed into a data record through the importing module, and the data record is stored in the database so as to complete data importing.
In the data import system in this embodiment, data import is different from data import performed after a template file for data import is created in an encoding environment, and this embodiment simplifies operations of service personnel, improves convenience for creating the template file for data import, and thus improves convenience for data import.
In addition, in the data import system in the embodiment, service personnel do not need to perform complex coding, so that the time for creating the template file for data import is reduced, and the work efficiency of data import is improved.
Meanwhile, in the embodiment, the data import system does not need a service person to create the template file in the coding environment, so that the data import template file is created without depending on the operation and technology of the service person, and the operability of data import is improved.
Based on the data importing system, an embodiment of the present invention provides a data importing method, which is shown in fig. 1, where fig. 1 is a schematic flow diagram of a first embodiment of the data importing method according to the present invention.
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.
In this embodiment, the data importing method is applied to the data importing system, and the data importing system includes: the system comprises an interactive interface for man-machine interaction with business personnel, a storage module, a creation module and an import module. The data importing method comprises the following steps:
step S10: receiving and storing a table creation rule configured by the service personnel based on the interactive interface through the storage module;
the business personnel can configure the form creating rule for creating the form file in the interactive interface provided by the data import system.
In this embodiment, the table creation rule configured by the business personnel based on the interactive interface is received and stored by the storage module. In particular embodiments, the table creation rules may include: field encoding, field description, and field rules.
Further, in the specific implementation, business personnel can bind the form creation rule with information such as business scenes and form types through an interactive interface. And business personnel can determine the required form creation rule according to the required form type, the business scene of data import and the like.
Step S20: when a template generation instruction is received, calling a target rule corresponding to the template generation instruction from the storage module through the creation module to generate a template file, so that the service personnel can use the template file to perform data entry to obtain a data file;
in this embodiment, when a template generation instruction is received, the creation module calls a target rule corresponding to the template generation instruction from the storage module to generate a template file, so that service personnel can use the template file to perform data entry.
In a specific embodiment, the template generation instruction may carry information corresponding to the table creation rule, for example, service scenario information or table type information of data import, so that the creation module invokes the target rule from the storage module based on the information.
Specifically, in an embodiment, after the template file is generated by the creating module, the generated information can be returned, so that after the template file is downloaded by a service person based on an interactive interface, the downloaded template file is used for data entry; in another embodiment, after the template file is generated by the creating module, the template file may be displayed through the interactive interface, so that business personnel may use the template file for data entry on the interactive interface, and the template file is not particularly limited herein and may be set according to actual requirements.
Step S30: receiving the data file uploaded by the service personnel based on the interactive interface;
after the business personnel use the template file to complete data entry to obtain the data file, the data file can be uploaded to conduct data import.
Specifically, in this embodiment, the service staff uploads the data file through the interactive interface. And receiving the data file uploaded by the service personnel based on the interactive interface.
Specifically, in an embodiment, when a service person downloads a template file based on an interactive interface and then uses the downloaded template file to perform data entry, the service person may upload the data file locally through an upload function provided by the interactive interface; in another embodiment, when the business personnel uses the template file to perform data entry on the interactive interface, the business personnel can save and upload the data file through a saving function provided by the interactive interface, which is not limited herein.
Step S40: and analyzing the data file into a data record through the import module, and storing the data record into a database.
In this embodiment, after receiving a data file uploaded by a service worker based on an interactive interface, the data file is parsed into data records by the import module, and the data records are stored in the database to complete data import.
Further, in an embodiment, analyzing the data file into the data records may be performed by splitting the data file into a plurality of subfiles and performing task analysis processing based on a multithreading concurrency technical idea; in another embodiment, the parsing data record may also be in a serial manner, which is not limited herein and may be set according to actual requirements.
Further, in one embodiment, before storing the data records in the database, the data records may be checked to determine whether each piece of data in the data records meets the target rule. When the data records are not verified, the data records are processed correspondingly so as to import correct data into the database. Thereby improving the accuracy of data processing.
In the embodiment, a form creation rule configured by a business person based on an interactive interface is received and stored through a storage module, when a template generation instruction is received, a target rule corresponding to the template generation instruction is called from the storage module through a creation module to generate a template file, so that the business person can use the template file to perform data entry to obtain and upload a data file, the data file uploaded by the business person based on the interactive interface is received, the data file is analyzed into a data record through an import module, and the data record is stored in a database.
The method and the device are different from the method and the device for data import after the template file for data import is created in the coding environment, so that the operation of business personnel is simplified, the convenience for creating the template file for data import is improved, and the convenience for data import is improved.
In addition, in the data import system in the embodiment, service personnel do not need to perform complex coding, so that the time for creating the template file for data import is reduced, and the work efficiency of data import is improved.
Meanwhile, in the embodiment, the data import system does not need service personnel to create the template file in the coding environment, so that the operation and the technology of the service personnel are not depended when the template file for data import is created, and the operability of data import is improved.
Further, a second embodiment of the data importing method of the present invention is proposed based on the first embodiment, and in this embodiment, in the step S40: before the step of storing the data record into the database, the data importing method further comprises:
step S50: for any target record in the data records, verifying the target record through the import module;
in the embodiment, after the data file is analyzed into the data records through the import module, the data records are verified through the import module before being stored in the database, so that the accuracy of each data in the data records is ensured.
Specifically, in this embodiment, for any data record in the data records (hereinafter referred to as a target record for distinction), the import module verifies the target record.
The target rules may include: field encoding, field description, and field rules. In a specific embodiment, it may be detected whether each data of the target record conforms to the field rule, so as to verify the target record; or determining the padding flag, the available value range, and the threshold range of each data based on the field rule, and checking the target record by detecting whether each data of the target record conforms to the respective corresponding padding flag, the available value range, and the threshold range, which is not limited herein.
Step S60: and if the verification is passed, executing the step of storing the data record into a database.
In this embodiment, after the import module verifies the target record, if the verification passes, it is determined that the target record is not abnormal, and at this time, the import module may store the data record into the database.
It should be noted that, in this embodiment, before the data record is stored in the database through the import module, the data record is verified through the import module, and when it is determined that the verification is passed, the data record is stored in the database, so that the accuracy of the data record in the database is ensured, and the practicability of data import is improved. Meanwhile, the situation that data processing needs to be carried out again due to abnormal data in the data processing process is avoided, and the working efficiency of data processing is improved.
Further, in some possible embodiments, in the above step S50: after the import module verifies the target record, the data import method further includes:
step S70: if the verification fails, displaying a verification failure record through the interactive interface so that the business personnel can modify and upload the verification failure record;
in this embodiment, a data record that fails to be verified (hereinafter, referred to as a verification failure record to indicate distinction) is processed to import a correct data record.
Specifically, after the target record is verified through the import module, if the verification fails, it is determined that the target record is abnormal, and at this time, the target record is a verification failure record.
In this embodiment, when the target record is determined to be the verification failure record, the verification failure record is displayed through the interactive interface, so that the service personnel can modify and upload the verification failure record.
In a specific implementation manner, a downloading function can be provided through a service interaction interface, so that service personnel can download the verification failure record, and locally modify and upload the verification failure record; the verification failure record may also be displayed through an interactive interface, so that service personnel modify and upload the verification failure record on the interactive interface, which is not limited herein.
Step S80: and receiving the modified verification failure record, taking the verification failure record as the target record, and executing the step of verifying the target record through the import module.
In this embodiment, the modified verification failure record is received, the verification failure record is used as a target record, the step of verifying the target record through the importing module is executed, and after the verification is determined to be passed, the target record is imported into the database. Therefore, the accuracy of data records in the database is ensured, and the practicability of data import is improved.
It should be noted that, in this embodiment, the check failure record is displayed through the interactive interface, so that the service personnel can modify and upload the check failure record, and the modified check failure data uploaded by the service personnel is imported into the database through the import module.
Further, in some possible embodiments, the table creation rule includes: field encoding, field description and field rules, wherein the field rules include rules characterizing the required fill identifier, data type, data sequence and fill format, in the above step S40: the step of verifying the target record through the import module comprises the following steps:
step S401: detecting whether each data in the target record meets the target rule or not through the import module;
in this embodiment, the target record is verified by verifying whether each data in the target record satisfies the field rule.
Specifically, in this embodiment, it is checked by the import module whether each data in the target record meets the field rule. In a specific embodiment, the data may be checked for the required fill identifier, the data type, the data sequence, and the fill format, all of which are consistent with the field rule. If they are consistent, it may be determined that the data satisfies the field rule.
Step S402: if yes, determining that the target record passes verification;
in this embodiment, whether each data in the target record meets the field rule is checked by the import module. And if the target record is satisfied, determining that the target record passes verification. At this time, the data record can be stored in the database through the import module.
Step S403: if not, determining that the target record is not verified.
In this embodiment, whether each data in the target record meets the field rule is checked by the import module. If the difference is not satisfied, determining that data which does not satisfy the field rule exists in the target record, namely, the target record is not verified. At this time, a target record of the verification failure, that is, a verification failure record, can be displayed through the interactive interface, so that the business personnel can modify and upload the verification failure record, and the modified verification failure data uploaded by the business personnel is imported into the database through the import module.
It should be noted that before the data record is stored in the database through the import module, the data record is verified through the import module, and when the verification is determined to be passed, the data record is stored in the database, so that the accuracy of the data record in the database is ensured, and the practicability of data import is improved.
Further, in some possible embodiments, in the above step S40: before the data file is parsed into the data records by the importing module, the data importing method further includes:
step S90: detecting whether the data volume of the data file is larger than a preset threshold value or not;
in the data importing process, the importing of data files with a large data amount is usually involved, in this embodiment, the data analysis processing is disassembled into atomic steps, and the task analysis processing is performed based on a multithreading concurrency technical idea, so as to improve the efficiency of data analysis.
Specifically, in this embodiment, it is detected whether the data amount of the data file is greater than a preset threshold. The preset threshold may be set according to actual requirements, and is not limited herein.
Step A10: if the data volume of the data file is less than or equal to the preset threshold value, the step of analyzing the data file into a data record through the import module is executed;
in this embodiment, if it is detected that the data amount of the data file is less than or equal to the preset threshold, the data file may not be split, and the data file is directly parsed into the data records by the importing module, so as to import the data records into the database.
Step A20: if the data volume of the data file is larger than the preset threshold value, the data file is divided into a plurality of sub-files containing the preset data volume through the import module, and each sub-file is analyzed into the data record.
If the data volume of the data file is detected to be larger than the preset threshold value, the data file is determined to be too large, and the analysis speed and the importing speed of the importing module may be influenced. At this time, the data file may be split into multiple subfiles including a preset data amount by the import module, and each of the subfiles may be parsed into the data record. In a specific embodiment, the plurality of subfiles can be synchronously analyzed through the importing module, so that the analyzing speed is increased, and the work efficiency of data importing is improved.
Further, in an embodiment, when the data file is split into multiple subfiles, data columns with the same data identifier may be placed in the same subfile.
It should be noted that, in this embodiment, after the data file of which the data amount exceeds the preset threshold is split into multiple subfiles, the subfiles are analyzed through the import module, and compared with the current serial analysis mode, this embodiment achieves an analysis speed improvement, so that the work efficiency of data import is improved.
In the embodiment, before the data records are stored in the database through the import module, the data records are verified through the import module, and when the verification is determined to pass, the data records are stored in the database, so that the accuracy of the data records in the database is ensured, and the practicability of data import is improved. Meanwhile, the situation that data processing needs to be carried out again due to abnormal data in the data processing process is avoided, and the working efficiency of data processing is improved.
Further, based on the first and/or second embodiments, a third embodiment of the data importing method according to the present invention is provided, in this embodiment, the data importing system further includes a processing module, and the data importing method further includes:
step A30: sending the address information of the data record in the database to a data processing terminal through the processing module, so that the data processing terminal reads the data record from the database based on the address information, performs data processing on the data record to generate a processing result and returns the processing result;
in this embodiment, the data import system further includes a processing module, and the processing module may be configured to interact with the processing terminal. After the data import is finished through the import module, the information can be sent to the data processing terminal through the processing module, so that the data processing terminal can call data records to perform data analysis and data processing based on the information.
Specifically, in this embodiment, after the data import is completed by the import module, the address information of the data recorded in the database may be sent to the data processing terminal through the processing module.
And after receiving the address information, the data processing terminal reads the data record from the database based on the address information, and processes the data record to generate a processing result. In a specific implementation mode, if the data processing terminal determines that each data record has no abnormality in the data processing process, a normal result representing that the processing is finished is generated; if the data processing terminal finds an abnormal data record in the data processing process, for example, the data in the data record does not meet the corresponding target rule, at this time, the data processing terminal generates an abnormal result representing that the abnormal data record exists.
And after the data processing terminal generates a processing result, returning the processing result to the data import system, wherein the processing result is a normal result representing that the data processing is finished or an abnormal result representing that an abnormal data record exists.
Step A40: receiving the processing result, and if the processing result is determined to be an abnormal result representing that an abnormal data record exists, acquiring the abnormal data record from the database through the processing module based on the abnormal result;
and after the data processing terminal finishes data processing on the data records, the data processing terminal can generate a processing result and return the processing result to the data import system.
In this embodiment, a processing result is received, where the processing result is a normal result representing that data processing is completed or an abnormal result representing that an abnormal data record exists.
And after receiving the processing result, if the processing result is determined to be an abnormal result representing that the abnormal data record exists, acquiring the abnormal data record from the database through the processing module based on the abnormal result.
In a specific embodiment, the exception result may include exception information characterizing the exception data record, where the exception information may include a data identifier of the exception data record and address information of the data record used for currently performing data processing. And acquiring an abnormal data record corresponding to the abnormal information from the database through the processing module based on the abnormal information contained in the abnormal result.
Step A50: displaying the abnormal data record through the interactive interface so that the business personnel can modify and upload the abnormal data record;
and after the abnormal data record is acquired through the processing module, the abnormal data record can be displayed through the interactive interface. And the abnormal data records are modified and uploaded by service personnel.
In a specific implementation, business personnel can modify the abnormal data records on the interactive interface through the interactive interface; or a downloading function provided by the interactive interface is provided, so that business personnel can download the abnormal data records based on the interactive interface and upload the abnormal data records after locally modifying the abnormal data records. Specifically, reference may be made to steps S20 to S30, which are not described herein again.
Step A60: and receiving the modified abnormal data record, and storing the modified abnormal data record into the database through the import module so that the data processing terminal can process data based on the modified abnormal data record.
In this embodiment, an abnormal data record uploaded by a user based on an interactive interface is received. And storing the modified abnormal data record into a database through an import module. And the data processing terminal processes the data based on the modified abnormal data record.
In a specific embodiment, the modified abnormal data record is stored in a database through an import module, wherein the modified abnormal data record can be stored in the same position as the abnormal data record; the modified abnormal data record may also be stored in a different location from the abnormal data record, which is not limited herein.
In a specific embodiment, the specific process involved in the data processing performed by the data terminal based on the modified abnormal data record may refer to step a30 to step a60, which is not described herein in detail.
Further, in some possible embodiments, in the step a60: the step of storing the modified abnormal data record into the database through the import module comprises the following steps:
step A601: reading a data identifier of the abnormal data record through the import module;
in this embodiment, the modified abnormal data record is stored in the same position as the abnormal data record through the import module.
Each data record corresponds to a unique data identifier, e.g., a data name to which the data corresponds, from which the different data records can be distinguished.
In this embodiment, the data identifier of the abnormal data record is read by the import module to locate the abnormal data record in the database based on the data identifier and the address information. It will be appreciated that the data identifiers of the exception data record and the modified exception data record are consistent.
Step A602: replacing, by the import module, the modified exception data record with the modified exception data record based on the address information and the data identifier to store the modified exception data record in the database.
And after the data identifier of the abnormal data record is read through the import module, replacing the abnormal data record with the modified abnormal data record through the import module based on the address information and the data identifier so as to store the modified abnormal data record into the database.
It should be noted that, by replacing the modified abnormal data record with the abnormal data record, the modified abnormal data record is stored in the database, and compared with the case that the modified abnormal data record is stored in the database at an address different from the abnormal data record, the data management system of this embodiment does not need to read corresponding data records from multiple addresses in the database during data processing, thereby reducing the complexity of data processing.
In this embodiment, the data import system further includes a processing module. In the embodiment, the processing module sends the address information of the data record in the database to the data processing terminal, receives the processing result sent by the data processing terminal, and if the processing result is determined to be an abnormal result representing that the abnormal data record exists, the processing module obtains the abnormal data record from the database based on the abnormal result, and displays the abnormal data record through the interactive interface. And receiving the abnormal data record modified by the service personnel, and storing the modified abnormal data record into a database through an import module.
Compared with the existing data processing process, when abnormal data records occur, business personnel are required to modify and re-import the data files so as to process the data. According to the embodiment, the operation of service personnel is simplified, and the working efficiency of data processing is improved.
In addition, the present invention also provides a data importing apparatus, which is applied to a data importing system including: the system comprises an interaction interface, a storage module, a creation module and an import module, wherein the interaction interface is used for performing human-computer interaction with service personnel;
referring to fig. 2, fig. 2 is a schematic diagram of functional modules of a data importing apparatus according to an embodiment of the present invention. The data importing device of the invention comprises:
the first calling module 10 is used for receiving and storing the table creation rule configured by the service personnel based on the interactive interface through the storage module;
the second calling module 20 is configured to, when a template generation instruction is received, call, by the creating module, a target rule corresponding to the template generation instruction from the storage module to generate a template file, so that the service staff uses the template file to perform data entry to obtain a data file and upload the data file;
a third calling module 30, configured to receive the data file uploaded by the service staff based on the interactive interface;
and the fourth calling module 40 is configured to parse the data file into a data record through the import module, and store the data record in a database.
Further, the fourth invoking module 40 is further configured to:
for any target record in the data records, detecting whether each data in the target record meets the target rule through the import module;
and if the data records are satisfied, the step of storing the data records into a database is executed.
Further, the fourth invoking module 40 is further configured to:
if the difference is met, displaying a verification failure record through the interactive interface so that the service personnel can modify and upload the verification failure record;
and receiving the modified verification failure record, taking the verification failure record as the target record, and executing the step of verifying the target record through the import module.
Further, the table creation rule includes: the field encoding method comprises the following steps of field encoding, field description and field rules, wherein the field rules comprise rules for representing mandatory filling identification, data types, data sequences and filling formats;
the fourth invoking module 40 is further configured to:
detecting whether each data in the target record meets the field rule or not through the import module;
if so, determining that the target record passes verification;
if not, determining that the target record is not verified.
Further, the fourth invoking module 40 is further configured to:
detecting whether the data volume of the data file is larger than a preset threshold value or not;
if the data volume of the data file is less than or equal to the preset threshold, the step of analyzing the data file into a data record through the import module is executed;
if the data volume of the data file is larger than the preset threshold value, the data file is split into multiple subfiles containing the preset data volume through the import module, and each subfile is analyzed into the data record.
Further, the data importing system further includes a processing module, and the data importing apparatus further includes a fifth calling module, where the fifth calling module is configured to:
sending the address information of the data record in the database to a data processing terminal through the processing module, so that the data processing terminal reads the data record from the database based on the address information, performs data processing on the data record to generate a processing result and returns the processing result;
receiving the processing result, and if the processing result is determined to be an abnormal result representing that an abnormal data record exists, acquiring the abnormal data record from the database through the processing module based on the abnormal result;
displaying the abnormal data record through the interactive interface so that the business personnel can modify and upload the abnormal data record;
and receiving the modified abnormal data record, and storing the modified abnormal data record into the database through the import module so that the data processing terminal can process data based on the modified abnormal data record.
Further, the fifth invoking module is further configured to:
reading a data identifier of the abnormal data record through the import module;
replacing, by the import module, the modified exception data record with the modified exception data record based on the address information and the data identifier to store the modified exception data record in the database.
The steps of the data importing method are implemented when each functional module of the data importing apparatus runs.
In addition, the invention also provides data import equipment. Referring to fig. 3, fig. 3 is a schematic structural diagram of a data importing apparatus according to an embodiment of the present invention. The data import device in the embodiment of the present invention may specifically be a device that locally runs a data import system.
As shown in fig. 3, the data importing apparatus according to the embodiment of the present invention may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., a Wi-Fi interface).
A memory 1005 is provided on the data importing apparatus main body, and the memory 1005 stores a program that realizes a corresponding operation when executed by the processor 1001. The memory 1005 is also used to store parameters for use by the data import apparatus. The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001 described previously.
Those skilled in the art will appreciate that the data import device configuration shown in fig. 3 does not constitute a limitation of the data import device and may include more or less components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 3, a memory 1005, which is a storage medium, may include therein an operating system, a network processing module, a user interface module, and a data import program.
In the data import apparatus shown in fig. 3, the processor 1001 may be configured to call a data import program stored in the memory 1005, and perform the following operations:
receiving and storing a table creation rule configured by the service personnel based on the interactive interface through the storage module;
when a template generation instruction is received, calling a target rule corresponding to the template generation instruction from the storage module through the creation module to generate a template file, so that the service personnel can use the template file to perform data entry to obtain a data file and upload the data file;
receiving the data file uploaded by the service personnel based on the interactive interface;
and analyzing the data file into a data record through the import module, and storing the data record into a database.
Further, the processor 1001 may be further configured to call a data importing program stored in the memory 1005, and perform the following operations:
for any target record in the data records, detecting whether each data in the target record meets the target rule through the import module;
and if the data records are satisfied, executing the step of storing the data records into a database.
Further, the processor 1001 may be further configured to call a data importing program stored in the memory 1005, and perform the following operations:
if the difference is met, displaying a verification failure record through the interactive interface so that the service personnel can modify and upload the verification failure record;
and receiving the modified verification failure record, taking the verification failure record as the target record, and executing the step of verifying the target record through the import module.
Further, the processor 1001 may be further configured to call a data importing program stored in the memory 1005, and perform the following operations:
detecting whether each data in the target record meets the field rule or not through the import module;
if yes, determining that the target record passes verification;
if not, determining that the target record is not verified.
Further, the processor 1001 may be further configured to call a data importing program stored in the memory 1005, and perform the following operations:
detecting whether the data volume of the data file is larger than a preset threshold value or not;
if the data volume of the data file is less than or equal to the preset threshold value, the step of analyzing the data file into a data record through the import module is executed;
if the data volume of the data file is larger than the preset threshold value, the data file is divided into a plurality of sub-files containing the preset data volume through the import module, and each sub-file is analyzed into the data record.
Further, the processor 1001 may be further configured to call a data importing program stored in the memory 1005, and perform the following operations:
sending the address information of the data record in the database to a data processing terminal through the processing module, so that the data processing terminal reads the data record from the database based on the address information, performs data processing on the data record to generate a processing result and returns the processing result;
receiving the processing result, and if the processing result is determined to be an abnormal result representing that an abnormal data record exists, acquiring the abnormal data record from the database through the processing module based on the abnormal result;
displaying the abnormal data record through the interactive interface so that the business personnel can modify and upload the abnormal data record;
and receiving the modified abnormal data record, and storing the modified abnormal data record into the database through the import module so that the data processing terminal can process data based on the modified abnormal data record.
Further, the processor 1001 may be further configured to call a data importing program stored in the memory 1005, and perform the following operations:
reading a data identifier of the abnormal data record through the import module;
replacing, by the import module, the modified exception data record with the modified exception data record based on the address information and the data identifier to store the modified exception data record in the database.
In addition, the invention also provides a computer readable storage medium. Referring to fig. 4, fig. 4 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention. The computer readable storage medium has stored thereon a data import program, which when executed by a processor implements the steps of the data import method as described above.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of other like elements in a process, method, article, or system comprising the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM/RAM, magnetic disk, optical disk) as mentioned above and includes instructions for enabling a data importing device (which may be a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes performed by the present invention or directly or indirectly applied to other related technical fields are also included in the scope of the present invention.
Claims (10)
1. A data import method, wherein the data import method is applied to a data import system, and the data import system includes: the system comprises an interaction interface, a storage module, a creation module and an import module, wherein the interaction interface is used for performing human-computer interaction with service personnel;
the data import method comprises the following steps:
receiving and storing a table creation rule configured by the service personnel based on the interactive interface through the storage module;
when a template generation instruction is received, calling a target rule corresponding to the template generation instruction from the storage module through the creation module to generate a template file, so that the service personnel can use the template file to perform data entry to obtain a data file and upload the data file;
receiving the data file uploaded by the service personnel based on the interactive interface;
and analyzing the data file into a data record through the import module, and storing the data record into a database.
2. The data import method of claim 1, wherein prior to the step of storing the data records in the database, the data import method further comprises:
for any target record in the data records, verifying the target record through the import module;
and if the verification is passed, executing the step of storing the data record into a database.
3. The data import method of claim 2, wherein after the step of verifying the target record by the import module, the data import method further comprises:
if the verification fails, displaying a verification failure record through the interactive interface so that the business personnel can modify and upload the verification failure record;
and receiving the modified verification failure record, taking the verification failure record as the target record, and executing the step of verifying the target record through the import module.
4. The data import method of claim 3, wherein the table creation rule comprises: the field encoding method comprises the following steps of field encoding, field description and field rules, wherein the field rules comprise rules for representing mandatory filling identification, data types, data sequences and filling formats;
the step of verifying the target record through the import module includes:
detecting whether each data in the target record meets the field rule or not through the import module;
if so, determining that the target record passes verification;
if not, determining that the target record is not verified.
5. The data import method of claim 1, wherein prior to the parsing the data file into data records by the import module, the data import method further comprises:
detecting whether the data volume of the data file is larger than a preset threshold value or not;
if the data volume of the data file is less than or equal to the preset threshold value, the step of analyzing the data file into a data record through the import module is executed;
if the data volume of the data file is larger than the preset threshold value, the data file is divided into a plurality of sub-files containing the preset data volume through the import module, and each sub-file is analyzed into the data record.
6. The data import method of any of claims 1 to 5, wherein the data import system further comprises a processing module;
the data importing method further comprises the following steps:
sending the address information of the data record in the database to a data processing terminal through the processing module, so that the data processing terminal reads the data record from the database based on the address information, performs data processing on the data record to generate a processing result and returns the processing result;
receiving the processing result, and if the processing result is determined to be an abnormal result representing that an abnormal data record exists, acquiring the abnormal data record from the database through the processing module based on the abnormal result;
displaying the abnormal data record through the interactive interface so that the business personnel can modify and upload the abnormal data record;
and receiving the modified abnormal data record, and storing the modified abnormal data record into the database through the import module so that the data processing terminal can process data based on the modified abnormal data record.
7. The data importing method according to claim 6, wherein the step of storing the modified abnormal data record into the database through the importing module includes:
reading a data identifier of the abnormal data record through the import module;
replacing, by the import module, the modified exception data record with the modified exception data record based on the address information and the data identifier to store the modified exception data record in the database.
8. A data import apparatus, wherein the data import apparatus is applied to a data import system, and the data import system includes: the system comprises an interaction interface, a storage module, a creation module and an import module, wherein the interaction interface is used for performing human-computer interaction with service personnel;
the data importing apparatus includes:
the first calling module is used for receiving and storing a table creation rule configured by the service personnel based on the interactive interface through the storage module;
the second calling module is used for calling a target rule corresponding to the template generation instruction from the storage module through the creating module to generate a template file when the template generation instruction is received, so that the business personnel can use the template file to perform data entry to obtain a data file and upload the data file;
the third calling module is used for receiving the data file uploaded by the service personnel based on the interactive interface;
and the fourth calling module is used for analyzing the data file into a data record through the importing module and storing the data record into a database.
9. A data import apparatus, characterized in that the data import apparatus comprises: a memory, a processor, and a data import program stored on the memory and executable on the processor, the data import program configured to implement the steps of the data import method of any of claims 1 to 7.
10. A computer-readable storage medium, on which a data import program is stored, which, when executed by a processor, implements the steps of the data import method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211300021.6A CN115617780A (en) | 2022-10-21 | 2022-10-21 | Data import method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211300021.6A CN115617780A (en) | 2022-10-21 | 2022-10-21 | Data import method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115617780A true CN115617780A (en) | 2023-01-17 |
Family
ID=84864594
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211300021.6A Pending CN115617780A (en) | 2022-10-21 | 2022-10-21 | Data import method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115617780A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115982137A (en) * | 2023-03-17 | 2023-04-18 | 鲁班(北京)电子商务科技有限公司 | Data name and database tabulation generation method and system |
| CN117608536A (en) * | 2023-12-20 | 2024-02-27 | 杭州明佑电子有限公司 | Gap data online template customization and supplementary recording system and method thereof |
-
2022
- 2022-10-21 CN CN202211300021.6A patent/CN115617780A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115982137A (en) * | 2023-03-17 | 2023-04-18 | 鲁班(北京)电子商务科技有限公司 | Data name and database tabulation generation method and system |
| CN115982137B (en) * | 2023-03-17 | 2023-08-01 | 鲁班(北京)电子商务科技有限公司 | Data name and database table building generation method and system |
| CN117608536A (en) * | 2023-12-20 | 2024-02-27 | 杭州明佑电子有限公司 | Gap data online template customization and supplementary recording system and method thereof |
| CN117608536B (en) * | 2023-12-20 | 2024-05-28 | 杭州明佑电子有限公司 | Gap data online template customization and supplementary recording system and method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108876121B (en) | Work order processing method and device, computer equipment and storage medium | |
| CN115617780A (en) | Data import method, device, equipment and storage medium | |
| CN109977012B (en) | Joint debugging test method, device, equipment and computer readable storage medium of system | |
| CN113127347A (en) | Interface testing method, device, equipment and readable storage medium | |
| CN110688305B (en) | Test environment synchronization method, device, medium and electronic equipment | |
| CN112579455B (en) | Automatic interface testing method and device, electronic equipment and storage medium | |
| CN112069073A (en) | Test case management method, terminal and storage medium | |
| CN113704110A (en) | Automatic testing method and device for user interface | |
| CN111679851A (en) | Demand code management method, apparatus, system and computer readable storage medium | |
| CN111737152A (en) | Method and device for realizing WEB automatic test by inputting data through webpage | |
| CN110543429A (en) | Test case debugging method and device and storage medium | |
| CN114265737A (en) | System migration integrity detection method and device and electronic equipment | |
| CN113238935A (en) | Application testing method, system, device, medium, and computer program product | |
| CN113377669A (en) | Automatic testing method and device, computer equipment and storage medium | |
| CN111538542B (en) | System configuration method and related device | |
| CN113434405A (en) | Method and device for determining test file, storage medium and electronic device | |
| CN112433935A (en) | Test method, test device, electronic equipment and storage medium | |
| KR101252358B1 (en) | Apparatus and method for testing plc command | |
| CN111026669A (en) | Test log management method, test log management device, and storage medium | |
| CN112416648A (en) | Data verification method and device | |
| CN114371866A (en) | Version reconfiguration test method, device and equipment of service system | |
| CN114936152A (en) | Application testing method and device | |
| CN110737426B (en) | Program block creating method, program block creating device, computer equipment and storage medium | |
| CN114356396A (en) | Service publishing method and device adaptive to multiple services | |
| CN113377682A (en) | Service testing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |