CN110781231A - Batch import method, device, equipment and storage medium based on database - Google Patents
Batch import method, device, equipment and storage medium based on database Download PDFInfo
- Publication number
- CN110781231A CN110781231A CN201910884666.0A CN201910884666A CN110781231A CN 110781231 A CN110781231 A CN 110781231A CN 201910884666 A CN201910884666 A CN 201910884666A CN 110781231 A CN110781231 A CN 110781231A
- Authority
- CN
- China
- Prior art keywords
- data
- imported
- import
- database
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to the field of artificial intelligence, and discloses a batch import method, a batch import device, a batch import equipment and a batch import storage medium based on a database, which are used for avoiding all rollback of data in the same batch and improving the data import efficiency. The method comprises the following steps: acquiring a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is service data needing to be imported into a database; determining a data format of a target data set; mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported; importing a plurality of instances into a database according to a data format; monitoring the import processes of the multiple instances through a preset function, and storing the import results into the corresponding instances; when at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm.
Description
Technical Field
The invention relates to the field of artificial intelligence, in particular to a batch import method, a batch import device, a batch import equipment and a batch import storage medium based on a database.
Background
With the rapid development of internet technology and the need of business expansion, in some specific scenes, batch data needs to be imported into a database for storage, if the batch data is uploaded, the database performance is consumed by interacting with the database for many times in a short time by adopting a method of submitting and storing one piece of data, and the page response time is very long, foreground operation is stopped, and background database operation is still continued. Therefore, in general, when there is more than one data volume, a batch submission mode is adopted, however, due to the defect of the architectural design, the exception generated in the persistence process cannot be captured in time, so that the page cannot feed back the background processing result, and the operator cannot know whether the final data import is successful or not.
In the existing scheme, verification and feedback can be performed only before data import, and when data persistence fails due to database exception in the persistence process of partial data in batch imported data, the batch data import is rolled back completely and the persistence fails.
Disclosure of Invention
The invention provides a batch import method, a batch import device, equipment and a storage medium based on a database, which are used for eliminating abnormal data and re-importing normal data of the same batch of data when the batch import data is abnormal, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
A first aspect of an embodiment of the present invention provides a batch import method based on a database, including: acquiring a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data needing to be imported into a database; determining a data format of the target data set; mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported; importing the plurality of instances into the database according to the data format; monitoring the import processes of the multiple instances through a preset function, and storing the import results into the corresponding instances; when at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm.
Optionally, in a first implementation manner of the first aspect of the embodiment of the present invention, mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, where each instance includes one piece of data to be imported, includes: calling a preset class to split each piece of data to be imported to obtain N arrays of each piece of data to be imported, wherein the preset class comprises N attributes, and N is a positive integer; mapping the N data groups of each piece of data to be imported into the N attributes of the preset class respectively; adding a target attribute in the preset class, wherein the target attribute is used for recording abnormal information of the database; and generating an instance corresponding to each piece of data to be imported according to the N +1 attributes of the preset class to obtain a plurality of instances, wherein each instance comprises N arrays of the data to be imported and the database exception information.
Optionally, in a second implementation manner of the first aspect of the embodiment of the present invention, the importing, according to the data format, the multiple instances into the database includes: determining at least one target import template according to the data format; importing the plurality of instances into the database according to the at least one target import template.
Optionally, in a third implementation manner of the first aspect of the embodiment of the present invention, the determining at least one target import template according to the data format includes: judging whether the data format is a table format or a comma separated value format; if the data format is a table format or a comma separated value format, reading a plurality of title names in the data to be imported; and selecting at least one import template from preset candidate import templates as a target import template, wherein the target import template comprises the plurality of title names.
Optionally, in a fourth implementation manner of the first aspect of the embodiment of the present invention, the monitoring, by using a preset function, the import processes of the multiple instances, and storing the import results in the corresponding instances includes: calling a preset monitoring function to monitor the import process of the multiple instances; judging whether each array in each example is successfully imported or not; if any array fails to be imported in the target instance, marking the array which fails to be imported and determining the array as an abnormal array; generating exception information, wherein the exception information is used for indicating a marked exception array; and writing the exception information into the target attribute of the target instance.
Optionally, in a fifth implementation manner of the first aspect of the embodiment of the present invention, when at least one instance is abnormal in an importing process, importing, by using a preset classification algorithm, normal data in other instances in the target data set into the database, where the importing includes: when at least one example is abnormal in the importing process, calling a preset logistic regression model method to search an abnormal example which fails to be imported; deleting the exception instance; normal data in the remaining instances is imported to the database.
Optionally, in a sixth implementation manner of the first aspect of the embodiment of the present invention, before the obtaining a target data set, where the target data set includes multiple pieces of data to be imported, and the data to be imported is business data that needs to be imported into a database, the method further includes: acquiring original data and verifying to obtain verified target data; determining the target data as the target data set.
A second aspect of the embodiments of the present invention provides a batch importing apparatus based on a database, including: the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a target data set, the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is service data needing to be imported into a database; a first determining unit for determining a data format of the target data set; the mapping unit is used for mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, and each instance comprises one piece of data to be imported; a first importing unit, configured to import the multiple instances into the database according to the data format; the monitoring unit is used for monitoring the import processes of the multiple instances through a preset function and storing the import results into the corresponding instances; and the second import unit is used for importing the normal data in other examples in the target data set into the database through a preset binary classification algorithm when at least one example is abnormal in the import process.
Optionally, in a first implementation manner of the second aspect of the embodiment of the present invention, the mapping unit is specifically configured to: calling a preset class to split each piece of data to be imported to obtain N arrays of each piece of data to be imported, wherein the preset class comprises N attributes, and N is a positive integer; mapping the N data groups of each piece of data to be imported into the N attributes of the preset class respectively; adding a target attribute in the preset class, wherein the target attribute is used for recording abnormal information of the database; and generating an instance corresponding to each piece of data to be imported according to the N +1 attributes of the preset class to obtain a plurality of instances, wherein each instance comprises N arrays of the data to be imported and the database exception information.
Optionally, in a second implementation manner of the second aspect of the embodiment of the present invention, the first importing unit includes: the determining module is used for determining at least one target import template according to the data format; and the importing module is used for importing the multiple instances into the database according to the at least one target importing template.
Optionally, in a third implementation manner of the second aspect of the embodiment of the present invention, the determining module is specifically configured to: judging whether the data format is a table format or a comma separated value format; if the data format is a table format or a comma separated value format, reading a plurality of title names in the data to be imported; and selecting at least one import template from preset candidate import templates as a target import template, wherein the target import template comprises the plurality of title names.
Optionally, in a fourth implementation manner of the second aspect of the embodiment of the present invention, the monitoring unit is specifically configured to: calling a preset monitoring function to monitor the import process of the multiple instances; judging whether each array in each example is successfully imported or not; if any array fails to be imported in the target instance, marking the array which fails to be imported and determining the array as an abnormal array; generating exception information, wherein the exception information is used for indicating a marked exception array; and writing the exception information into the target attribute of the target instance.
Optionally, in a fifth implementation manner of the second aspect of the embodiment of the present invention, the second import unit is specifically configured to: when at least one example is abnormal in the importing process, calling a preset logistic regression model method to search an abnormal example which fails to be imported; deleting the exception instance; normal data in the remaining instances is imported to the database.
Optionally, in a sixth implementation manner of the second aspect of the embodiment of the present invention, the batch importing apparatus based on a database further includes: the acquisition and verification unit is used for acquiring original data and verifying the original data to obtain target data passing the verification; a second determining unit configured to determine the target data as the target data set.
A third aspect of the embodiments of the present invention provides a batch import apparatus based on a database, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the batch import method based on the database according to any of the above embodiments when executing the computer program.
A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the computer program implements the steps of the database-based batch import method according to any of the above-mentioned embodiments.
According to the technical scheme provided by the embodiment of the invention, a target data set is obtained, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data needing to be imported into a database; determining a data format of a target data set; mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported; importing a plurality of instances into a database according to a data format; monitoring the import processes of the multiple instances through a preset function, and storing the import results into the corresponding instances; when at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm. According to the embodiment of the invention, when the batch import data is abnormal, the abnormal data is removed and the normal data of the same batch of data is imported again, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
Drawings
FIG. 1 is a diagram of an embodiment of a batch import database-based method according to the embodiment of the present invention;
FIG. 2 is a diagram of another embodiment of a batch import database-based method according to the embodiment of the present invention;
FIG. 3 is a diagram of an embodiment of a batch database-based importing apparatus according to an embodiment of the present invention;
FIG. 4 is a diagram of another embodiment of a batch database-based importing apparatus according to an embodiment of the present invention;
fig. 5 is a diagram of an embodiment of a batch import device based on a database in an embodiment of the present invention.
Detailed Description
The invention provides a batch import method, a batch import device, equipment and a storage medium based on a database, which are used for eliminating abnormal data and re-importing normal data of the same batch of data when the batch import data is abnormal, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
In order to make the technical field of the invention better understand the scheme of the invention, the embodiment of the invention will be described in conjunction with the attached drawings in the embodiment of the invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, a flowchart of a batch import method based on a database according to an embodiment of the present invention specifically includes:
101. and acquiring a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is service data needing to be imported into the database.
The server acquires a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data needing to be imported into the database.
It should be noted that, the multiple pieces of data to be imported in the target data set are preprocessed service data, and the preprocessing process may include service data verification and other manners, specifically may include deduplication processing, error screening and other manners, and is not limited herein.
It is to be understood that the execution subject of the present invention may be a batch importing apparatus based on a database, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
102. The data format of the target data set is determined.
The server determines the data format of the target data set. The data format includes a table Excel format and a Comma Separated Value (CSV) format, and may also include other formats, which are not limited herein. For ease of understanding, the present invention is illustrated in Excel format.
103. And mapping each piece of data to be imported into a plurality of attributes of the preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported.
The server maps each piece of data to be imported to a plurality of attributes of the preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported. Specifically, the server calls a preset class to split each piece of data to be imported to obtain N arrays of each piece of data to be imported, wherein the preset class comprises N attributes, and N is a positive integer; the server maps N data groups of each piece of data to be imported into N attributes of a preset class respectively; the server adds a target attribute in a preset class, wherein the target attribute is used for recording abnormal information of the database; the server generates an instance corresponding to each piece of data to be imported according to the N +1 attributes of the preset class to obtain a plurality of instances, wherein each instance comprises N arrays of the data to be imported and database exception information. For example, the preset class batchInsert () includes a table name, a table field group $ keys to be inserted, e.g., [ 'name', 'age' ], and table data $ data () to be inserted, e.g., [ [ [ 'Tom',30], [ 'Jane',20], [ 'Linda',25] ].
It should be noted that, in the embodiment of the present invention, the preset class is uploadDto, which is a preset class, the class has a plurality of attributes, each attribute corresponds to a column in an uploaded Excel document, each line of data in the document is an individual uploadDto instance, the number of attributes is generally greater than the number of columns in the document, for example, the uploadDto therein is mostly provided with an errorInfo attribute for recording information of a check failure or a persistence failure.
104. The plurality of instances are imported into a database according to a data format.
The server imports a plurality of instances into a database according to the data format. Specifically, the server determines at least one target import template according to the data format; the server imports a plurality of instances into a database according to at least one target import template.
The process that the server determines at least one target import template according to the data format comprises the following steps: the server judges whether the data format is a table format or a comma separated value format; if the data format is a table format or a comma separated value format, the server reads a plurality of title names in the data to be imported; the server selects at least one import template from preset candidate import templates as a target import template, and the target import template comprises a plurality of title names.
It should be noted that the import template may be an Excel template or a CSV template, and the import template may also be a template in another format, which is not limited herein. The Excel template is a value of which column of the system corresponds to each column of the appointed Excel, and the corresponding modes comprise two modes, namely sequential matching and column name matching. Wherein, according to the sequence matching, the first column (column A) is appointed to correspond to which column of the system database, and the column B is appointed to correspond to which column. The Excel template may have no title lines or may have multiple title lines. Where column name matching refers to matching the names of designated columns, this case requires a titled row. The match indicates whether the space, the sign, in the title is omitted, because some templates will use a star to indicate that the column must be lost, and there may be a space. The CSV template is a comma-separated column-to-column template, and is usually a plain text file. It will be appreciated that a CSV file may be opened in Excel.
105. And monitoring the import process of the multiple instances through a preset function, and storing the import result into the corresponding instance.
The server monitors the import process of the multiple instances through a preset function, and stores the import result into the corresponding instance. Specifically, the server calls a preset monitoring function to monitor the import process of the multiple instances; the server judges whether each array in each instance is successfully imported or not; if any array fails to be imported in the target instance, the server marks the array which fails to be imported and determines the array as an abnormal array; the server generates abnormal information, and the abnormal information is used for indicating the marked abnormal array; the server writes the exception information into the target attributes of the target instance.
When the server and the database interact, the server analyzes the uploaded file into one uploadDto instance according to rows, a set of a plurality of uploaddtos is finally formed, then a batchInsert function is called, the entry of the function is a list set, all uploadDto instances are stored in the list set, the entry of the function is converted into an insert statement by the batchInsert function, the entry is submitted to the database for execution after the conversion is finished, and a commit (commit) command is executed to finish the storage after the execution is finished.
106. When at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm.
When at least one instance is abnormal in the importing process, the server imports normal data in other instances in the target data set into the database through a preset binary classification algorithm. For example,
specifically, when an exception occurs in the importing process of at least one instance, the server calls a preset logistic regression model method to search for an exception instance failed in importing; the server deletes the abnormal instance; the server imports the normal data in the remaining instances into the database.
For example, when a field of a piece of data of an example is too long, the field appears in the importing process, when the executing of the Insert statement fails, that is, an exception is encountered, a set of processing mechanism is introduced, the server determines which piece of data has a problem and records the data, a preset bisection algorithm is called to locate the abnormal data, the data importing is continued, half of the data without the problem is submitted each time of bisection, the data with the problem is continued to be halved, the target data is found and then recorded, and the target data is removed until all the data to be imported are imported.
It should be noted that, in this embodiment, the possible database level exceptions include: violation of unique constraints (some field values require no duplication with existing data in the database), violation of foreign key constraints (the value of a field must be one of a set of a field value in another table), database join failures, field overloads, data type of a field and database column type mismatch, etc.
It can be understood that if an exception related to the database occurs, the system can throw the exception, the exception is not captured and processed after being thrown, the operation is stopped and cannot be continued, if the exception is captured and not processed, the database is not inserted clearly and successfully, and an operator mistakenly regards that the database is inserted successfully because the exception is not received. Therefore, in this embodiment, the data of the insertion failure is found out, and the reason of the failure is recorded, which is actually processed after the exception is captured, so that the worker can know which data was successful and which data failed.
According to the embodiment of the invention, when the batch import data is abnormal, the abnormal data is removed and the normal data of the same batch of data is imported again, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
Referring to fig. 2, another flowchart of a batch import method based on a database according to an embodiment of the present invention specifically includes:
201. and acquiring original data and verifying to obtain verified target data.
And the server acquires the original data and checks the original data to obtain the target data passing the check. For example, if the first list company ID of the uploaded Excel document is specified and is a mandatory field, the Excel document is verified. If the company ID of a certain row of data in the uploaded Excel document is not filled, the system must list the data as abnormal data, and use the data filled with the company ID in other rows as target data.
202. The target data is determined as a target data set.
The server determines the target data as a target data set. And the server rearranges the data to obtain a target data set.
203. And acquiring a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is service data needing to be imported into the database.
The server acquires a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data needing to be imported into the database.
It should be noted that, the multiple pieces of data to be imported in the target data set are preprocessed service data, and the preprocessing process may include service data verification and other manners, specifically may include deduplication processing, error screening and other manners, and is not limited herein.
It is to be understood that the execution subject of the present invention may be a batch importing apparatus based on a database, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
204. The data format of the target data set is determined.
The server determines the data format of the target data set. The data format includes a table Excel format and a Comma Separated Value (CSV) format, and may also include other formats, which are not limited herein. For ease of understanding, the present invention is illustrated in Excel format.
205. And mapping each piece of data to be imported into a plurality of attributes of the preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported.
The server maps each piece of data to be imported to a plurality of attributes of the preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported. Specifically, the server calls a preset class to split each piece of data to be imported to obtain N arrays of each piece of data to be imported, wherein the preset class comprises N attributes, and N is a positive integer; the server maps N data groups of each piece of data to be imported into N attributes of a preset class respectively; the server adds a target attribute in a preset class, wherein the target attribute is used for recording abnormal information of the database; the server generates an instance corresponding to each piece of data to be imported according to the N +1 attributes of the preset class to obtain a plurality of instances, wherein each instance comprises N arrays of the data to be imported and database exception information. For example, the preset class batchInsert () includes a table name, a table field group $ keys to be inserted, e.g., [ 'name', 'age' ], and table data $ data () to be inserted, e.g., [ [ [ 'Tom',30], [ 'Jane',20], [ 'Linda',25] ].
It should be noted that, in the embodiment of the present invention, the preset class is uploadDto, which is a preset class, the class has a plurality of attributes, each attribute corresponds to a column in an uploaded Excel document, each line of data in the document is an individual uploadDto instance, the number of attributes is generally greater than the number of columns in the document, for example, the uploadDto therein is mostly provided with an errorInfo attribute for recording information of a check failure or a persistence failure.
206. And determining at least one target import template according to the data format.
The server determines at least one target import template according to the data format. Specifically, the server judges whether the data format is a table format or a comma separated value format; if the data format is a table format or a comma separated value format, the server reads a plurality of title names in the data to be imported; the server selects at least one import template from preset candidate import templates as a target import template, and the target import template comprises a plurality of title names.
207. Importing the plurality of instances into a database according to at least one target import template.
The server imports a plurality of instances into a database according to at least one target import template.
It should be noted that the import template may be an Excel template or a CSV template, and the import template may also be a template in another format, which is not limited herein. The Excel template is a value of which column of the system corresponds to each column of the appointed Excel, and the corresponding modes comprise two modes, namely sequential matching and column name matching. Wherein, according to the sequence matching, the first column (column A) is appointed to correspond to which column of the system database, and the column B is appointed to correspond to which column. The Excel template may have no title lines or may have multiple title lines. Where column name matching refers to matching the names of designated columns, this case requires a titled row. The match indicates whether the space, the sign, in the title is omitted, because some templates will use a star to indicate that the column must be lost, and there may be a space. The CSV template is a comma-separated column-to-column template, and is usually a plain text file. It will be appreciated that a CSV file may be opened in Excel.
208. And monitoring the import process of the multiple instances through a preset function, and storing the import result into the corresponding instance.
The server monitors the import process of the multiple instances through a preset function, and stores the import result into the corresponding instance. Specifically, the server calls a preset monitoring function to monitor the import process of the multiple instances; the server judges whether each array in each instance is successfully imported or not; if any array fails to be imported in the target instance, the server marks the array which fails to be imported and determines the array as an abnormal array; the server generates abnormal information, and the abnormal information is used for indicating the marked abnormal array; the server writes the exception information into the target attributes of the target instance.
When the server and the database interact, the server analyzes the uploaded file into one uploadDto instance according to rows, a set of a plurality of uploaddtos is finally formed, then a batchInsert function is called, the entry of the function is a list set, all uploadDto instances are stored in the list set, the entry of the function is converted into an insert statement by the batchInsert function, the entry is submitted to the database for execution after the conversion is finished, and a commit (commit) command is executed to finish the storage after the execution is finished.
209. When at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm.
When at least one instance is abnormal in the importing process, the server imports normal data in other instances in the target data set into the database through a preset binary classification algorithm. For example,
specifically, when an exception occurs in the importing process of at least one instance, the server calls a preset logistic regression model method to search for an exception instance failed in importing; the server deletes the abnormal instance; the server imports the normal data in the remaining instances into the database.
For example, when a field of a piece of data of an example is too long, the field appears in the importing process, when the executing of the Insert statement fails, that is, an exception is encountered, a set of processing mechanism is introduced, the server determines which piece of data has a problem and records the data, a preset bisection algorithm is called to locate the abnormal data, the data importing is continued, half of the data without the problem is submitted each time of bisection, the data with the problem is continued to be halved, the target data is found and then recorded, and the target data is removed until all the data to be imported are imported.
It should be noted that, in this embodiment, the possible database level exceptions include: violation of unique constraints (some field values require no duplication with existing data in the database), violation of foreign key constraints (the value of a field must be one of a set of a field value in another table), database join failures, field overloads, data type of a field and database column type mismatch, etc.
It can be understood that if an exception related to the database occurs, the system can throw the exception, the exception is not captured and processed after being thrown, the operation is stopped and cannot be continued, if the exception is captured and not processed, the database is not inserted clearly and successfully, and an operator mistakenly regards that the database is inserted successfully because the exception is not received. Therefore, in this embodiment, the data of the insertion failure is found out, and the reason of the failure is recorded, which is actually processed after the exception is captured, so that the worker can know which data was successful and which data failed.
According to the embodiment of the invention, when the batch import data is abnormal, the abnormal data is removed and the normal data of the same batch of data is imported again, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
With reference to fig. 3, the above description is made on a batch importing method based on a database in an embodiment of the present invention, and a batch importing apparatus based on a database in an embodiment of the present invention is described below, where an embodiment of the batch importing apparatus based on a database in an embodiment of the present invention includes:
an obtaining unit 301, configured to obtain a target data set, where the target data set includes multiple pieces of data to be imported, and the data to be imported is service data that needs to be imported into a database;
a first determining unit 302, configured to determine a data format of the target data set;
a mapping unit 303, configured to map each piece of data to be imported into multiple attributes of a preset class through the preset class, to generate multiple instances, where each instance includes one piece of data to be imported;
a first importing unit 304, configured to import the multiple instances into the database according to the data format;
a monitoring unit 305, configured to monitor the import processes of the multiple instances through a preset function, and store the import results in corresponding instances;
and the second importing unit 306 is configured to, when at least one instance is abnormal in the importing process, import normal data in other instances in the target data set into the database through a preset binary classification algorithm.
In the embodiment of the invention, when the batch import data is abnormal, the abnormal data is removed and the normal data of the same batch of data is imported again, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
Referring to fig. 4, another embodiment of the batch import apparatus based on a database according to the embodiment of the present invention includes:
an obtaining unit 301, configured to obtain a target data set, where the target data set includes multiple pieces of data to be imported, and the data to be imported is service data that needs to be imported into a database;
a first determining unit 302, configured to determine a data format of the target data set;
a mapping unit 303, configured to map each piece of data to be imported into multiple attributes of a preset class through the preset class, to generate multiple instances, where each instance includes one piece of data to be imported;
a first importing unit 304, configured to import the multiple instances into the database according to the data format;
a monitoring unit 305, configured to monitor the import processes of the multiple instances through a preset function, and store the import results in corresponding instances;
and the second importing unit 306 is configured to, when at least one instance is abnormal in the importing process, import normal data in other instances in the target data set into the database through a preset binary classification algorithm.
Optionally, the mapping unit 303 is specifically configured to:
calling a preset class to split each piece of data to be imported to obtain N arrays of each piece of data to be imported, wherein the preset class comprises N attributes, and N is a positive integer; mapping the N data groups of each piece of data to be imported into the N attributes of the preset class respectively; adding a target attribute in the preset class, wherein the target attribute is used for recording abnormal information of the database; and generating an instance corresponding to each piece of data to be imported according to the N +1 attributes of the preset class to obtain a plurality of instances, wherein each instance comprises N arrays of the data to be imported and the database exception information.
Optionally, the first import unit 304 includes:
a determining module 3041, configured to determine at least one target import template according to the data format;
an importing module 3042, configured to import the multiple instances into the database according to the at least one target import template.
Optionally, the determining module 3041 is specifically configured to:
judging whether the data format is a table format or a comma separated value format; if the data format is a table format or a comma separated value format, reading a plurality of title names in the data to be imported; and selecting at least one import template from preset candidate import templates as a target import template, wherein the target import template comprises the plurality of title names.
Optionally, the monitoring unit 305 is specifically configured to:
calling a preset monitoring function to monitor the import process of the multiple instances; judging whether each array in each example is successfully imported or not; if any array fails to be imported in the target instance, marking the array which fails to be imported and determining the array as an abnormal array; generating exception information, wherein the exception information is used for indicating a marked exception array; and writing the exception information into the target attribute of the target instance.
Optionally, the second import unit 306 is specifically configured to:
when at least one example is abnormal in the importing process, calling a preset logistic regression model method to search an abnormal example which fails to be imported; deleting the exception instance; normal data in the remaining instances is imported to the database.
Optionally, the batch importing apparatus based on the database further includes:
an obtaining and verifying unit 307, configured to obtain original data and perform verification, to obtain target data that passes verification;
a second determining unit 308, configured to determine the target data as the target data set.
In the embodiment of the invention, a target data set is obtained, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data needing to be imported into a database; determining a data format of a target data set; mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported; importing a plurality of instances into a database according to a data format; monitoring the import processes of the multiple instances through a preset function, and storing the import results into the corresponding instances; when at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm. In the embodiment of the invention, when the batch import data is abnormal, the abnormal data is removed and the normal data of the same batch of data is imported again, so that the data in the same batch are prevented from rolling back completely, and the data import efficiency is improved.
Fig. 3 to fig. 4 describe the database-based batch import apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the database-based batch import apparatus in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of a database-based batch import apparatus according to an embodiment of the present invention, where the database-based batch import apparatus 500 may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 501 (e.g., one or more processors) and a memory 509, and one or more storage media 508 (e.g., one or more mass storage devices) storing applications 507 or data 506. Memory 509 and storage medium 508 may be, among other things, transient storage or persistent storage. The program stored on the storage medium 508 may include one or more modules (not shown), each of which may include a series of instruction operations on a batch database-based import facility. Still further, the processor 501 may be configured to communicate with the storage medium 508 to execute a series of instruction operations in the storage medium 508 on the database-based batch import apparatus 500.
The database-based bulk import device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input-output interfaces 504, and/or one or more operating systems 505, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the database-based bulk import facility architecture shown in FIG. 5 does not constitute a limitation of database-based bulk import facilities, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components. The processor 501 may perform the functions of the acquisition unit 301, the first determination unit 302, the mapping unit 303, the first import unit 304, the monitoring unit 305, the second import unit 306, the acquisition check unit 307, and the second determination unit 308 in the above-described embodiments.
The following describes each component of the database-based batch import apparatus in detail with reference to fig. 5:
the processor 501 is a control center of the database-based batch import apparatus, and can perform processing according to a set database-based batch import method. The processor 501 connects various portions of the entire database-based batch import device using various interfaces and lines, performs various functions and processes data of the database-based batch import device by running or executing software programs and/or modules stored in the memory 509 and calling up the data stored in the memory 509, thereby achieving the vertebrae segmentation and disease symptom classification. The storage medium 508 and the memory 509 are carriers for storing data, in the embodiment of the present invention, the storage medium 508 may be an internal memory with a small storage capacity but a high speed, and the memory 509 may be an external memory with a large storage capacity but a low storage speed.
The memory 509 may be used to store software programs and modules, and the processor 501 executes various functional applications and data processing of the database-based batch import apparatus 500 by running the software programs and modules stored in the memory 509. The memory 509 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function (such as determining a data format of the target data set), and the like; the storage data area may store data (such as a plurality of instances, etc.) created according to the use of the database-based bulk import apparatus, and the like. Further, the memory 509 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. The batch import method program based on the database and the received data stream provided in the embodiment of the present invention are stored in the memory, and when they need to be used, the processor 501 calls the method program from the memory 509.
When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., compact disk), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A batch import method based on a database is characterized by comprising the following steps:
acquiring a target data set, wherein the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data needing to be imported into a database;
determining a data format of the target data set;
mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, wherein each instance comprises one piece of data to be imported;
importing the plurality of instances into the database according to the data format;
monitoring the import processes of the multiple instances through a preset function, and storing the import results into the corresponding instances;
when at least one instance is abnormal in the importing process, normal data in other instances in the target data set are imported into the database through a preset binary classification algorithm.
2. The batch import method based on database according to claim 1, wherein the mapping each piece of data to be imported into the plurality of attributes of the preset class through the preset class to generate a plurality of instances, each instance including one piece of data to be imported includes:
calling a preset class to split each piece of data to be imported to obtain N arrays of each piece of data to be imported, wherein the preset class comprises N attributes, and N is a positive integer;
mapping the N data groups of each piece of data to be imported into the N attributes of the preset class respectively;
adding a target attribute in the preset class, wherein the target attribute is used for recording abnormal information of the database;
and generating an instance corresponding to each piece of data to be imported according to the N +1 attributes of the preset class to obtain a plurality of instances, wherein each instance comprises N arrays of the data to be imported and the database exception information.
3. The database-based bulk import method of claim 1, wherein the importing the plurality of instances into the database according to the data format comprises:
determining at least one target import template according to the data format;
importing the plurality of instances into the database according to the at least one target import template.
4. The batch database-based import method of claim 3, wherein said determining at least one target import template according to the data format comprises:
judging whether the data format is a table format or a comma separated value format;
if the data format is a table format or a comma separated value format, reading a plurality of title names in the data to be imported;
and selecting at least one import template from preset candidate import templates as a target import template, wherein the target import template comprises the plurality of title names.
5. The batch import method based on the database according to claim 1, wherein the monitoring the import process of the multiple instances through a preset function and storing the import result in the corresponding instance comprises:
calling a preset monitoring function to monitor the import process of the multiple instances;
judging whether each array in each example is successfully imported or not;
if any array fails to be imported in the target instance, marking the array which fails to be imported and determining the array as an abnormal array;
generating exception information, wherein the exception information is used for indicating a marked exception array;
and writing the exception information into the target attribute of the target instance.
6. The batch import method based on the database according to claim 1, wherein when at least one instance is abnormal during the import process, importing normal data in other instances in the target dataset into the database through a preset binary classification algorithm, including:
when at least one example is abnormal in the importing process, calling a preset logistic regression model method to search an abnormal example which fails to be imported;
deleting the exception instance;
normal data in the remaining instances is imported to the database.
7. The batch import method based on the database according to any of claims 1 to 6, wherein before the obtaining of the target data set, the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is business data that needs to be imported into the database, the method further comprises:
acquiring original data and verifying to obtain verified target data;
determining the target data as the target data set.
8. A batch import device based on a database is characterized by comprising:
the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring a target data set, the target data set comprises a plurality of pieces of data to be imported, and the data to be imported is service data needing to be imported into a database;
a first determining unit for determining a data format of the target data set;
the mapping unit is used for mapping each piece of data to be imported into a plurality of attributes of a preset class through the preset class to generate a plurality of instances, and each instance comprises one piece of data to be imported;
a first importing unit, configured to import the multiple instances into the database according to the data format;
the monitoring unit is used for monitoring the import processes of the multiple instances through a preset function and storing the import results into the corresponding instances;
and the second import unit is used for importing the normal data in other examples in the target data set into the database through a preset binary classification algorithm when at least one example is abnormal in the import process.
9. A database-based batch import apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the database-based batch import method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the database-based batch import method according to any of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910884666.0A CN110781231B (en) | 2019-09-19 | 2019-09-19 | Database-based batch import method, device, equipment and storage medium |
| PCT/CN2019/119105 WO2021051627A1 (en) | 2019-09-19 | 2019-11-18 | Database-based batch importing method, apparatus and device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910884666.0A CN110781231B (en) | 2019-09-19 | 2019-09-19 | Database-based batch import method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781231A true CN110781231A (en) | 2020-02-11 |
| CN110781231B CN110781231B (en) | 2023-07-21 |
Family
ID=69384248
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910884666.0A Active CN110781231B (en) | 2019-09-19 | 2019-09-19 | Database-based batch import method, device, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110781231B (en) |
| WO (1) | WO2021051627A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581210A (en) * | 2020-04-30 | 2020-08-25 | 深圳壹账通智能科技有限公司 | Data importing method and system |
| CN112035459A (en) * | 2020-09-01 | 2020-12-04 | 中国银行股份有限公司 | Data format conversion method and device |
| CN112231329A (en) * | 2020-10-14 | 2021-01-15 | 北京人大金仓信息技术股份有限公司 | Method, device, equipment and readable storage medium for inserting data into database |
| CN112579717A (en) * | 2020-12-07 | 2021-03-30 | 泰康保险集团股份有限公司 | Method and device for displaying user address on electronic map |
| CN112632132A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Method, device and equipment for processing abnormal import data |
| CN113127428A (en) * | 2021-04-30 | 2021-07-16 | 深圳壹账通智能科技有限公司 | Data batch import method and device and related equipment |
| CN112231329B (en) * | 2020-10-14 | 2024-04-26 | 北京人大金仓信息技术股份有限公司 | Method, apparatus, device and readable storage medium for inserting data into database |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113468258A (en) * | 2021-07-21 | 2021-10-01 | 浙江大华技术股份有限公司 | Heterogeneous data conversion method and device and storage medium |
| CN116932252B (en) * | 2023-09-18 | 2024-01-26 | 北京冠群信息技术股份有限公司 | Asynchronous task compensation method and device based on batch data import pipeline |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002052431A1 (en) * | 2000-12-21 | 2002-07-04 | Fulltilt Solutions, Inc. | Method and system for importing data |
| CN106598979A (en) * | 2015-10-15 | 2017-04-26 | 北京国双科技有限公司 | Data writing method and device |
| CN107463661A (en) * | 2017-07-31 | 2017-12-12 | 小草数语（北京）科技有限公司 | The introduction method and device of data |
| CN108228730A (en) * | 2017-12-11 | 2018-06-29 | 深圳市买买提信息科技有限公司 | Data lead-in method, device, computer equipment and readable storage medium storing program for executing |
| CN108376171A (en) * | 2018-02-27 | 2018-08-07 | 平安科技（深圳）有限公司 | Method, apparatus, terminal device and the storage medium that big data quickly introduces |
| CN109656922A (en) * | 2018-12-19 | 2019-04-19 | 国网北京市电力公司 | Data processing method and device |
| CN110059491A (en) * | 2019-03-16 | 2019-07-26 | 平安城市建设科技(深圳)有限公司 | Data import monitoring method, device, equipment and readable storage medium storing program for executing |
| CN110162420A (en) * | 2019-04-26 | 2019-08-23 | 平安科技（深圳）有限公司 | Data assisted location method, device, computer equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101980187A (en) * | 2010-10-13 | 2011-02-23 | 中兴通讯股份有限公司 | Data warehousing method and device |
| CN108491475B (en) * | 2018-03-08 | 2020-03-31 | 平安科技（深圳）有限公司 | Data rapid batch import method, electronic device and computer readable storage medium |
| CN109144950B (en) * | 2018-07-20 | 2022-02-15 | 中国邮政储蓄银行股份有限公司 | Service data storage method and device |
-
2019
- 2019-09-19 CN CN201910884666.0A patent/CN110781231B/en active Active
- 2019-11-18 WO PCT/CN2019/119105 patent/WO2021051627A1/en active Application Filing
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2002052431A1 (en) * | 2000-12-21 | 2002-07-04 | Fulltilt Solutions, Inc. | Method and system for importing data |
| CN106598979A (en) * | 2015-10-15 | 2017-04-26 | 北京国双科技有限公司 | Data writing method and device |
| CN107463661A (en) * | 2017-07-31 | 2017-12-12 | 小草数语（北京）科技有限公司 | The introduction method and device of data |
| CN108228730A (en) * | 2017-12-11 | 2018-06-29 | 深圳市买买提信息科技有限公司 | Data lead-in method, device, computer equipment and readable storage medium storing program for executing |
| CN108376171A (en) * | 2018-02-27 | 2018-08-07 | 平安科技（深圳）有限公司 | Method, apparatus, terminal device and the storage medium that big data quickly introduces |
| CN109656922A (en) * | 2018-12-19 | 2019-04-19 | 国网北京市电力公司 | Data processing method and device |
| CN110059491A (en) * | 2019-03-16 | 2019-07-26 | 平安城市建设科技(深圳)有限公司 | Data import monitoring method, device, equipment and readable storage medium storing program for executing |
| CN110162420A (en) * | 2019-04-26 | 2019-08-23 | 平安科技（深圳）有限公司 | Data assisted location method, device, computer equipment and storage medium |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111581210A (en) * | 2020-04-30 | 2020-08-25 | 深圳壹账通智能科技有限公司 | Data importing method and system |
| CN112035459A (en) * | 2020-09-01 | 2020-12-04 | 中国银行股份有限公司 | Data format conversion method and device |
| CN112231329A (en) * | 2020-10-14 | 2021-01-15 | 北京人大金仓信息技术股份有限公司 | Method, device, equipment and readable storage medium for inserting data into database |
| CN112231329B (en) * | 2020-10-14 | 2024-04-26 | 北京人大金仓信息技术股份有限公司 | Method, apparatus, device and readable storage medium for inserting data into database |
| CN112579717A (en) * | 2020-12-07 | 2021-03-30 | 泰康保险集团股份有限公司 | Method and device for displaying user address on electronic map |
| CN112579717B (en) * | 2020-12-07 | 2024-02-13 | 泰康保险集团股份有限公司 | Method and device for displaying user address on electronic map |
| CN112632132A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Method, device and equipment for processing abnormal import data |
| CN112632132B (en) * | 2020-12-31 | 2024-04-12 | 中国农业银行股份有限公司 | Processing method, device and equipment for abnormal imported data |
| CN113127428A (en) * | 2021-04-30 | 2021-07-16 | 深圳壹账通智能科技有限公司 | Data batch import method and device and related equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781231B (en) | 2023-07-21 |
| WO2021051627A1 (en) | 2021-03-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781231A (en) | Batch import method, device, equipment and storage medium based on database | |
| CN109977110B (en) | Data cleaning method, device and equipment | |
| KR102177489B1 (en) | Method and device of searching index for sensor tag data | |
| US5842208A (en) | High performance recover/build index system by unloading database files in parallel | |
| CN107741903A (en) | Application compatibility method of testing, device, computer equipment and storage medium | |
| CN112199935A (en) | Data comparison method and device, electronic equipment and computer readable storage medium | |
| CN109522305B (en) | Big data deduplication method and device | |
| CN113918532A (en) | Portrait label aggregation method, electronic device and storage medium | |
| CN108121774B (en) | Data table backup method and terminal equipment | |
| US10089350B2 (en) | Proactive query migration to prevent failures | |
| US11301426B1 (en) | Maintaining stable record identifiers in the presence of updated data records | |
| CN114676126B (en) | Database-based data verification method, device, equipment and storage medium | |
| EP2213066A2 (en) | Acquisition and expansion of storage area network interoperation relationships | |
| CN110851437A (en) | Storage method, device and equipment | |
| CN113468076A (en) | Application program exception testing method, device, equipment and storage medium | |
| CN114371866A (en) | Version reconfiguration test method, device and equipment of service system | |
| CN112596948B (en) | Database cluster data backup method, device, equipment and storage medium | |
| CN110990475B (en) | Batch task inserting method and device, computer equipment and storage medium | |
| US9613051B2 (en) | Data processing method, information processing apparatus, and recording medium | |
| CN117453153B (en) | File storage method, device, terminal and medium based on flush rule | |
| CN112347095B (en) | Data table processing method, device and server | |
| JP5718256B2 (en) | System performance analysis apparatus, system performance analysis method, and system performance analysis program | |
| CN111459949B (en) | Data processing method, device and equipment for database and index updating method | |
| CN115827478A (en) | Code viewing method and device, computer equipment and storage medium | |
| CN115858493A (en) | Data migration method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |