CN115718777A - Data batch import method, system, computer device and storable medium - Google Patents
Data batch import method, system, computer device and storable medium Download PDFInfo
- Publication number
- CN115718777A CN115718777A CN202211369002.9A CN202211369002A CN115718777A CN 115718777 A CN115718777 A CN 115718777A CN 202211369002 A CN202211369002 A CN 202211369002A CN 115718777 A CN115718777 A CN 115718777A
- Authority
- CN
- China
- Prior art keywords
- data
- import
- template
- business object
- filling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a system, a computer device and a readable storage medium for batch import of data, wherein the method comprises the steps of loading an import template according to a business object model by a loading program, filling data into the import template as required, matching the corresponding relation of fields of business objects by a matching program according to the column names of the templates, taking the data of columns as the field values of the business objects, reading and converting the data in a multithreading way, and storing the data to a database in batch; according to the method, the import template is generated, and the data is filled in the import template according to the requirement, so that compared with the customized import template in the prior art, the customized template in the method is higher in flexibility and higher in applicability.
Description
Technical Field
The present invention relates to the field of data import technologies, and in particular, to a method, a system, a computer device, and a storage medium for batch importing of data.
Background
Data import is a common function of a system, and almost all systems should support the function, and is mainly used for initializing data in large quantities at the initial stage of the system or for the case that a large number of data lines need to be input.
The import template is designed in the development and design stage, the positions of the rows and columns are fixed, and the program reads and processes data according to the setting.
The general process of data import is that a user reads an Excel file according to a template format, then a program reads the file, the correspondence is carried out according to columns, and data are written into a system line by line;
1. CN102750368B discloses a database cluster data high-speed import method, which encapsulates a data import function into an independent data import drive component, realizes the data high-speed import function through the mutual cooperation of the data import drive component and an interactive control module in a control node of a database cluster, and transfers the use of CPU, memory and network resources by data import to a client, thereby liberating the control node and realizing the data import under the condition of being interfered by the database cluster as little as possible; meanwhile, an independent data import mode is adopted, parallel import can be conveniently realized by adding clients, and the import performance is improved, so that the requirement of an enterprise on high-speed data import is met;
2. CN110737636B discloses a data import method, device and equipment, which selects and reads files under a file transfer protocol directory for placing extensible markup language data files, analyzes extensible markup language data files with different structures into a file format matched with the file transfer protocol directory, creates a table structure for the analyzed extensible markup language data files, obtains quantitative data of the extensible markup language data files after the table structure is created, carries out accuracy verification on the extensible markup language data files after the table structure is created according to the quantitative data to obtain a verification result, imports extensible markup language data files with different structures, the verification result of which passes through the accuracy verification, into files under the same file transfer protocol directory, and can place the extensible markup language data files with different data structures into the files under the same file transfer protocol directory;
3. CN107145537B discloses a table data import method and system, which improve the verification efficiency and save the computer resources by grouping data with common points and then using a batch verification mode; one ORM object is shared by one group, so that the use efficiency of the memory is improved, and the memory consumption is reduced;
the above technical solution provides an improvement on data import, but the following problems still exist in the field of data batch import in the prior art:
1. taking the prior art "one" as an example, in the prior art, although parallel import can be conveniently realized by adding clients by adopting an independent data import mode, an import template of a business object is fixed, cannot be customized and is not flexible;
2. taking the prior art "one" as an example, in the prior art, although the extensible markup language data files with different data structures can be placed in the file under the same file transfer protocol directory, if the volume of the batch imported data is large, the processing efficiency is not high;
3. taking the prior art "one" as an example, in the prior art, although the verification efficiency is improved by grouping the data with common points and then using a batch verification mode, the computer resources are saved, but when the business system reaches a certain scale, a plurality of reference relations exist among business objects, and when data is imported in batch, the problem of data conversion of reference fields needs to be solved, which affects the import performance and efficiency;
therefore, a data batch import method, a data batch import system, a computer device and a storage medium are provided.
Disclosure of Invention
Embodiments of the present invention are directed to a method, a system, a computer device, and a storage medium for batch importing data, so as to solve or mitigate the technical problems in the prior art, and to provide at least one useful choice.
Interpretation of professional terms:
business objects refer to conceptual classes within a domain or visual representations of objects in the real world.
The bill head, one business object must have and only one bill head. A plurality of sub-document heads can be provided, wherein the document head and the sub-document head are 1:1 which is equivalent to a splitting table;
the bill body, a business object can have 0 or more bill bodies. The header and each body are 1:N. There is no association between the document bodies;
the technical scheme of the embodiment of the invention is realized as follows: the data batch import method comprises the following steps:
s1, loading a loading program to load an import template according to a business object model: the import template comprises all importable fields of the business object;
s2, filling data into the imported template according to the requirement: reserving fields (columns) needing to be imported according to needs, and filling data;
s3, matching the corresponding relation of the fields of the business object by the matching program according to the column name of the template, and taking the data of the column as the field value of the business object;
s4, multithreading reading-conversion data:
reading data: reading data from the template, and filling the data into a data entity of the business object;
converting data, and acquiring the id of the service object from the database for filling according to the code or the name of the service object;
s5, storing the data in a database in batches: and the storage processing unit acquires data in batches from the concurrent queue and stores the data in the database.
Further preferred is: and in the S1, a template is led in to flatten the bill head and the bill body.
Further preferred is: in S3, when the template configuration is imported, the method further includes setting a matching search field of the row, which is used for matching existing data and covering the data.
Further preferred is: in S4, when the step of reading data and the step of converting data are performed, parallel processing is performed by multi-thread execution.
Further preferred is: and after the data reading step and the data converting step, adding the data into a concurrent queue.
Further preferably: and in the S5, when the data is stored in the database, respectively filling the data into the data entity of the business object according to the bill head and the bill body.
Further preferred is: and when the data is stored, controlling the data processed in batches.
A data batch import system comprises:
the loading module is used for loading an import template according to the business object model, and the import template comprises all the import fields of the business object;
the filling module is used for filling data into the import template according to the requirement;
the matching module is used for matching the corresponding relation of the fields of the business object according to the column name of the template and taking the data of the column as the field value of the business object;
the reading module is used for filling the data entity of the business object from the import module;
the conversion module is used for acquiring the id of the service object from the database for filling according to the code or the name of the service object;
and the storage module is used for acquiring data in batches from the concurrent queue and storing the data in the database.
A computer device comprising a processor, a memory coupled to the processor, the memory having stored therein program instructions that, when executed by the processor, cause the processor to perform the steps of the data batch import method of any of the above.
A storage medium storing program instructions capable of implementing the data batch import method as claimed in any one of the preceding claims.
Due to the adoption of the technical scheme, the embodiment of the invention has the following advantages:
1. according to the method, the import template is generated, and the data is filled in the import template according to the requirement, so that compared with the customized import template in the prior art, the customized template in the method is higher in flexibility and higher in applicability.
2. According to the method and the device, the corresponding relation of the fields of the business object is matched according to the column name of the template by the matching program, and the data of the columns are used as the field values of the business object to be imported, so that the processing efficiency is higher under the condition that the batch imported data volume is large.
3. The data reading method and the data reading device read data from the template, respectively fill the data entity of the business object according to the bill head and the bill body, simultaneously acquire the id of the business object from the database for filling according to the code or the name of the business object in parallel, adopt the processing unit which divides reading and conversion into two parallel units and execute through multithreading, improve the processing efficiency and lead-in performance and efficiency.
The foregoing summary is provided for the purpose of description only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of the present invention will be readily apparent by reference to the drawings and the following detailed description.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the embodiments or technical descriptions will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a schematic diagram of a configuration of a import template according to the present invention;
FIG. 3 is a system block diagram of the input import system of the present invention;
FIG. 4 is a schematic diagram of the computer device of the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
The terms "first", "second" and "third" in this application are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or as implying a number of indicated technical features. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of the feature. In the description of the present application, "plurality" means at least two, e.g., two, three, etc., unless explicitly specifically limited otherwise. In the embodiment of the present application, all the directional indicators (such as upper, lower, left, right, front, and rear … …) are used only to explain the relative positional relationship between the components, the movement, and the like in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indicator is changed accordingly. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
Example one
As shown in fig. 1 to 4, an embodiment of the present invention provides a data batch import method, including the following steps:
s1, loading a import template according to a business object model by a loading program: the import template comprises all importable fields of the business object;
s2, filling data into the imported template according to the requirement: reserving fields (columns) needing to be imported according to needs, and filling data;
s3, configuring an import template: as shown in fig. 2, the matching program matches the correspondence of the fields of the business object according to the column name of the template, and takes the data of the column as the field value of the business object;
s4, multithreading reading-conversion data:
reading data: reading data from the template, and filling the data into a data entity of the business object;
converting data, and acquiring the id of the service object from the database for filling according to the code or name of the service object;
when the business system reaches a certain scale, a plurality of reference relations exist among the business objects. When data is imported in batches, the problem of data conversion of the reference fields needs to be solved, and the problems of the performance and the efficiency of import are influenced.
The method divides reading and conversion into two parallel processing units and executes the two parallel processing units through multiple threads, so that the processing efficiency is improved.
And reading data, reading the data from the template according to the configuration of the S3, and filling the data into the data entity of the business object according to the bill head and the bill body respectively.
Converting data, and acquiring the id of the service object from the database for filling according to the code or the name of the service object;
s5, storing data in a database in batches: and the storage processing unit acquires data in batches from the concurrent queue and stores the data in the database.
In this embodiment, specifically: and in the S1, a template is led in to flatten the bill head and the bill body.
In this embodiment, specifically: in S3, when the template configuration is imported, the method further includes setting a matching search field of the row, which is used for matching existing data and covering the data.
In this embodiment, specifically: in S4, when the step of reading data and the step of converting data are performed, parallel processing is performed by multi-thread execution.
In this embodiment, specifically: and after the data reading step and the data converting step, adding the data into a concurrent queue.
In this embodiment, specifically: and in the S5, when the data is stored in the database, filling the data into the data entity of the business object according to the bill head and the bill body.
In this embodiment, specifically: and when the data is stored, controlling the data processed in batches.
A data batch import system comprises:
the loading module is used for loading an import template according to the business object model, and the import template comprises all import fields of the business object;
the filling module is used for filling data into the import template according to the requirement;
the matching module is used for matching the corresponding relation of the fields of the business object according to the column name of the template and taking the data of the column as the field value of the business object;
the reading module is used for filling the data entity of the business object from the import module;
the conversion module is used for acquiring the id of the service object from the database for filling according to the code or the name of the service object;
and the storage module is used for acquiring data in batches from the concurrent queue and storing the data in the database.
For other details of the technical solution for implementing each module in the data batch import system in the above embodiment, reference may be made to the description of the data batch import method in the above embodiment, and details are not described here again.
It should be noted that, in the present specification, the embodiments are all described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the system-class embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
A computer device comprising a processor, a memory coupled to the processor, the memory having stored therein program instructions that, when executed by the processor, cause the processor to perform the steps of the data batch import method of any of the above.
The processor is an integrated circuit chip having signal processing capabilities. The processor may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
A storage medium, which stores program instructions capable of implementing the data batch import method described in any one of the above embodiments, wherein the program instructions may be stored in the storage medium in the form of a software product, and include instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or various media capable of storing program codes, or a computer device such as a computer, a server, a mobile phone, or a tablet. The server may be an independent server, or may be a cloud server that provides basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), and a big data and artificial intelligence platform.
Example two
The invention also provides another embodiment adopting the steps of the data importing method of the embodiment, which comprises the following specific steps:
(1) Generating import templates
Generating an import template according to the model of the business object, wherein the template comprises all the import fields of the business object, and the import fields are set by self-definition before being imported, for example, the field attribute is set as: commodities, basic information, a whole order, attribute details, a product sale organization, unit details and packaging details; and the template is led in to flatten the bill head and the bill body, so that the learning cost of data filling is reduced.
(2) On-demand data filling into import templates
On the basis of the step 1, reserving fields (columns) which need to be imported as required, and filling data;
such as: document fields such as commodities, basic information and the like are reserved;
(3) Configuration import template
After uploading the template file based on step 2, the system matches the corresponding relationship of the fields of the business object according to the column name of the template, and takes the data of the column as the field value of the business object, as shown in fig. 2:
importing a document field: commodity _ code, commodity _ name, commodity _ K3 main material Yun Lian ID, commodity _ K3 sub-material name, basic information _ standard suite scheme, basic information _ individual sale, basic information _ commodity detail description, whether commodity _ material auxiliary attribute is started, and whether commodity _ color auxiliary attribute is started;
and then, carrying out matching search fields according to the matching search basis, wherein the matching search basis comprises the following steps: the method comprises the steps of (integral order) searching basis, [ attribute detail ] searching, [ product sales organization ] searching, [ unit detail ] searching and [ packaging detail ] searching;
meanwhile, the matching search field of the row can be set for matching the existing data and covering the data.
(4) Multi-threaded read-convert data
When the business system reaches a certain scale, a plurality of reference relations exist among the business objects. When data is imported in batches, the problem of data conversion of the reference fields needs to be solved, and the performance and efficiency of import are affected.
The method divides reading and conversion into two parallel processing units and executes the two parallel processing units through multiple threads, so that the processing efficiency is improved.
And reading data, reading the data from the template according to the configuration in the step 3, and filling the data into the data entity of the business object according to the bill head and the bill body.
And converting data, and acquiring the id of the service object from the database for filling according to the code or the name of the service object.
(5) Bulk saving of data to a database
And (5) after the data processing in the step 4, adding the data into a concurrent queue, and obtaining data in batches from the concurrent queue by a storage processing unit and storing the data into a database.
In the storage processing unit, the data processed in batches can be controlled, and IO (input/output) blockage or deadlock of the database is avoided.
And finishing the import of batch data through the steps.
Compared with the customized import template in the existing method, the customized template in the method has higher flexibility and stronger applicability.
Compared with the prior method, the reading-conversion-saving is performed in sequence, and the efficiency is low. The method divides the three into three processing units, and after one unit finishes processing, the semi-finished products are handed over to the next unit for processing by analogy with a pipeline, and the efficiency can be improved by increasing the pipeline (threads).
In the several embodiments provided in the present application, it should be understood that the disclosed terminal, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is only a logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit. The above embodiments are merely examples and are not intended to limit the scope of the present disclosure, and all modifications, equivalents, and flow charts using the contents of the specification and drawings of the present disclosure or those directly or indirectly applied to other related technical fields are intended to be included in the scope of the present disclosure.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive various changes or substitutions within the technical scope of the present invention, and these should be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.
Claims (10)
1. A data batch import method is characterized by comprising the following steps:
s1, loading a loading program to load an import template according to a business object model: the import template comprises all the import fields of the business object;
s2, filling data into the import template according to the requirement: reserving fields needing to be imported according to needs, and filling data;
s3, matching the corresponding relation of the fields of the business object by the matching program according to the column name of the template, and taking the data of the column as the field value of the business object;
s4, multithreading reading-conversion data:
reading data: reading data from the template, and filling the data into a data entity of the business object;
converting data, and acquiring the id of the service object from the database for filling according to the code or name of the service object;
s5, storing the data in a database in batches: and the storage processing unit acquires data in batches from the concurrent queue and stores the data in the database.
2. The batch import method of data according to claim 1, wherein: and in the S1, a template is led in to flatten the bill head and the bill body.
3. The batch import method of data according to claim 1, wherein: in S3, when the template configuration is imported, the method further includes setting a matching search field of the row, which is used for matching existing data and covering the data.
4. The batch import method of data according to claim 1, wherein: in the step S4, when the data reading step and the data converting step are performed, the data are processed in parallel by multi-thread execution.
5. The batch import method of data according to claim 4, wherein: and after the data reading step and the data converting step, adding the data into a concurrent queue.
6. The batch import method of data according to claim 1, wherein: and in the S5, when the data is stored in the database, filling the data into the data entity of the business object according to the bill head and the bill body.
7. The batch import method of data according to claim 5, wherein: and when the data is stored, controlling the data processed in batches.
8. The batch import system of data according to any of claims 1 to 7, comprising:
the loading module is used for loading an import template according to the business object model, and the import template comprises all the import fields of the business object;
the filling module is used for filling data into the import template according to the requirement;
the matching module is used for matching the corresponding relation of the fields of the business object according to the column name of the template and taking the data of the column as the field value of the business object;
the reading module is used for filling the data entity of the business object from the import module;
the conversion module is used for acquiring the id of the service object from the database for filling according to the code or the name of the service object;
and the storage module is used for acquiring data in batches from the concurrent queue and storing the data in the database.
9. A computer device, characterized in that the computer device comprises a processor, a memory coupled to the processor, in which memory program instructions are stored which, when executed by the processor, cause the processor to carry out the steps of the data batch import method according to any of claims 1 to 7.
10. A storage medium storing program instructions capable of implementing the data batch import method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211369002.9A CN115718777A (en) | 2022-11-03 | 2022-11-03 | Data batch import method, system, computer device and storable medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211369002.9A CN115718777A (en) | 2022-11-03 | 2022-11-03 | Data batch import method, system, computer device and storable medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115718777A true CN115718777A (en) | 2023-02-28 |
Family
ID=85254716
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211369002.9A Pending CN115718777A (en) | 2022-11-03 | 2022-11-03 | Data batch import method, system, computer device and storable medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115718777A (en) |
-
2022
- 2022-11-03 CN CN202211369002.9A patent/CN115718777A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109947789B (en) | Method, device, computer equipment and storage medium for processing data of multiple databases | |
| CN112800095B (en) | Data processing method, device, equipment and storage medium | |
| CN111177113B (en) | Data migration method, device, computer equipment and storage medium | |
| CN107632869B (en) | Method and equipment for loading user interface of POS application | |
| CN106648569B (en) | Target serialization realization method and device | |
| CN110795697A (en) | Logic expression obtaining method and device, storage medium and electronic device | |
| CN112328551A (en) | Medical data analysis method, device, medium, and electronic device | |
| CN111858617A (en) | User searching method and device, computer readable storage medium and electronic equipment | |
| CN113254455B (en) | Dynamic configuration method and device of database, computer equipment and storage medium | |
| CN109597825B (en) | Rule engine calling method, device, equipment and computer readable storage medium | |
| CN114637866A (en) | Information management method and device for digital new media | |
| CN113515717B (en) | Navigation page updating method and device, electronic equipment and storage medium | |
| CN113901098A (en) | Template-based data export method, device, medium and equipment | |
| CN111444235A (en) | Django-based data serialization method and device, computer equipment and storage medium | |
| CN115718777A (en) | Data batch import method, system, computer device and storable medium | |
| CN113296912B (en) | Task processing method, device, system, storage medium and electronic equipment | |
| CN115114321A (en) | Dynamic query method and system | |
| CN110471708B (en) | Method and device for acquiring configuration items based on reusable components | |
| CN113704418A (en) | Customer service robot system, related method, device and equipment | |
| CN117271840B (en) | Data query method and device of graph database and electronic equipment | |
| CN117708384B (en) | Picture data storage method, device, equipment and storage medium based on JanusGraph | |
| CN118673234A (en) | Self-adaptive display method and system for dictionary data in power information application system | |
| CN116069865A (en) | Inventory sharing method, apparatus, computer device and storage medium | |
| CN117931910A (en) | Data storage method, device, equipment and storage medium | |
| CN118312512A (en) | Data import method, data export device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |