CN116431363A - Data import and export method and device based on message queue, medium and equipment - Google Patents

Data import and export method and device based on message queue, medium and equipment Download PDF

Info

Publication number
CN116431363A
CN116431363A CN202310364430.0A CN202310364430A CN116431363A CN 116431363 A CN116431363 A CN 116431363A CN 202310364430 A CN202310364430 A CN 202310364430A CN 116431363 A CN116431363 A CN 116431363A
Authority
CN
China
Prior art keywords
task
service module
file
import
imported
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310364430.0A
Other languages
Chinese (zh)
Inventor
韩双成
段其朴
张峰
王彦功
张悦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Software Technology Co Ltd
Original Assignee
Inspur Software Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Software Technology Co Ltd filed Critical Inspur Software Technology Co Ltd
Priority to CN202310364430.0A priority Critical patent/CN116431363A/en
Publication of CN116431363A publication Critical patent/CN116431363A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G06F9/546—Message passing systems or structures, e.g. queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention provides a data import and export method and device based on a message queue, a medium and equipment. The method comprises the following steps: uploading a file to be imported to an application service module, so that the application service module uploads the file to be imported to an object storage service module for storage, generates an import task, and pushes the import task to a message queue; wherein the file to be exported comprises structured data and the attachment; and after the task service module monitors that a new task exists in the message queue, if the task is an import task, downloading a file to be imported corresponding to the import task from the object storage service module, and importing the file to be imported into a database. The embodiment of the invention can reduce the waiting of users, prevent the overtime of the network, reduce the load of the server, improve the high availability of the application and also accord with the idea of micro-service.

Description

Data import and export method and device based on message queue, medium and equipment
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for importing and exporting data based on a message queue, a medium, and a device.
Background
The data import and export are basic functions of the service system, the data volume in each service system is larger and larger at present, the data types are also more and more complex, and a lot of data is often required to be collected and arranged down-line and then imported into the service system or be distributed to related personnel in an off-line mode when exported from the service system. In addition to the structured data, the imported and exported data often have attachments such as pictures, word files, excel files, compressed files and the like, and the attachments need to be processed while the structured data is processed.
Since a large amount of import and export operations take a large amount of time, and the target web application pursues a fast response, each link gateway has response time limit, so that it is difficult to conduct normal import and export of data in a short time. In order to more efficiently implement data import and export, reduce user latency and improve system throughput, it is necessary to provide a new import and export scheme.
Disclosure of Invention
Aiming at least one technical problem, the embodiment of the invention provides a data import and export method, a device, a medium and equipment based on a message queue.
According to a first aspect, a method for importing and exporting data based on a message queue provided by an embodiment of the present invention includes:
uploading a file to be imported to an application service module, so that the application service module uploads the file to be imported to an object storage service module for storage, generates an import task, and pushes the import task to a message queue; wherein the file to be exported comprises structured data and the attachment;
and after the task service module monitors that a new task exists in the message queue, if the task is an import task, downloading a file to be imported corresponding to the import task from the object storage service module, and importing the file to be imported into a database.
According to a second aspect, a data import and export device based on a message queue provided by an embodiment of the present invention includes:
the first import module is used for uploading the file to be imported to the application service module so that the application service module can upload the file to be imported to the object storage service module for storage, an import task is generated, and the import task is pushed to the message queue; wherein the file to be exported comprises structured data and the attachment;
and the second import module is used for downloading a file to be imported corresponding to the import task from the object storage service module and importing the file to be imported into a database when the task service module monitors that a new task exists in the message queue and if the task is the import task.
According to a third aspect, embodiments of the present invention provide a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method provided by the first aspect.
According to a fourth aspect, a computing device provided by an embodiment of the present invention includes a memory and a processor, where the memory stores executable code, and the processor implements the method provided by the first aspect when executing the executable code.
According to the data import and export method, device, medium and equipment based on the message queue, the import task is managed through the message queue by establishing the import task, and the import operation is realized based on the object storage service module. By the method, the file can be imported asynchronously, after the user performs related operations at the front end, the rear end server sequentially executes the tasks in the message queue, the user does not need to execute a large number of manual importing operations, the problems of slow importing speed of the multi-metadata and overtime of the network are solved, concurrency of long links of data processing is reduced, stability of a service system is guaranteed, and user experience is improved. Namely, the embodiment of the invention can reduce the waiting of users, prevent the overtime of the network, reduce the load of the server, improve the high availability of the application and also accord with the idea of micro-service.
Drawings
Fig. 1 is a flow chart of a data import/export method based on a message queue according to an embodiment of the invention.
Detailed Description
In a first aspect, an embodiment of the present invention provides a method for importing and exporting data based on a message queue, referring to fig. 1, the method includes steps S110 to S120 as follows:
s110, uploading a file to be imported to an application service module, so that the application service module uploads the file to be imported to an object storage service module for storage, generates an import task, and pushes the import task to a message queue; wherein the file to be exported comprises structured data and the attachment;
and S120, after the task service module monitors that a new task exists in the message queue, if the task is an import task, downloading a file to be imported corresponding to the import task from the object storage service module, and importing the file to be imported into a database. After the new task, if the task is the import task, then
In an actual scenario, a user organizes structured data and attachments into files to be imported at the front end, and then uploads the files to be imported to the back end server. And after receiving the file to be imported, the back-end server uploads the file to be imported to an application service module, wherein the application service module is a module for providing application service. And the application service module uploads the file to be imported to the object storage service module, and the object storage service module is used for providing object storage service. The application service module then generates an import task and then sends the import task to the message queue. When the task service module monitors that a new import task is added in the message queue, if no other tasks are executed currently, the new import task is directly executed, otherwise, all the tasks in the message queue are sequentially executed according to the sequence. When the task service module executes an import task, firstly, downloading a file to be imported corresponding to the import task from the object service module, and importing the file to be imported into the database, thereby completing the import work of the file.
In one embodiment, the method provided by the embodiment of the invention can further include:
and writing back the state of the import task so as to enable a front-end user to check whether the import work of the file to be imported is successfully completed.
For example, after completing an import task, the task service module writes back the status of the import task, where status refers to whether the import task was completed successfully, and writing back refers to feeding back to the front end. Thus, the user at the front end can know whether the importing work of the file to be imported is successfully completed or not.
In one embodiment, the method provided by the embodiment of the invention can further include:
and uploading the attachment to be imported into the object storage service module.
For example, after completing an import task, the task service module may upload the attachment in the import file to the object storage service module in order to facilitate a subsequent export of the file including the attachment by the user.
In one embodiment, the method provided by the embodiment of the invention can further include:
and storing the import task into the database.
For example, the application service module stores the import task in the database, so that when a problem occurs in the message queue, the import task can be obtained from the database, and the normal operation of the import work is ensured.
In one embodiment, the method provided by the embodiment of the invention can further include:
receiving a data query condition of a user;
sending the data query condition to the application service module so that the application service module generates an export task and pushes the export task to a message queue;
and when the task service module monitors that the message queue has a new task, if the task is a exporting task, inquiring corresponding structured data from the database according to the data inquiry condition corresponding to the exporting task, inquiring corresponding accessories from the object storage service module, generating a file to be exported by the structured data and the accessories, and uploading the file to be exported to the object storage service module so as to enable a user to download the file from the object storage service module at the front end.
In an actual scenario, a user sets a data query condition at the front end, and when the back end server receives the data query condition, the data query condition is sent to an application service module on the back end server, and the application service module generates an export task and then pushes the export task to a message queue. When the task service module monitors that the message queue receives a new task, if the task is a exporting task, inquiring corresponding structured data from a database according to a data inquiring condition corresponding to the exporting task, inquiring an accessory corresponding to the structured data from the object storage service module, organizing the structured data and the accessory into an exporting file, and uploading the exporting file to the object service module, so that a user can download the exporting file at the front end.
In one embodiment, the method provided by the embodiment of the invention can further include:
and writing back the state of the export task and the download address so as to ensure that a front-end user checks whether the export work of the export file is successfully completed or not, and downloading according to the download address when the export work is successfully exported.
For example, the task service module writes back the status of the export task and the download address, i.e. feeds back to the front end, so that the front end user can know whether the export task is successfully executed, and if the export task is successfully executed, the export task can be downloaded from the object service module according to the download address.
In one embodiment, the method provided by the embodiment of the invention can further include:
and storing the export task to the database.
For example, the task service module stores the export tasks in the database, so that when a problem occurs in the message queue, the export tasks which need to be executed can be known from the database, and the export work is ensured to be normally performed.
With reference to table 1 below, the various information about the import task can be understood:
TABLE 1
Figure BDA0004166145020000051
Figure BDA0004166145020000061
Referring to Table 2, it can be appreciated that the various information of the export task:
TABLE 2
Figure BDA0004166145020000062
In the importing process, firstly, a file to be imported is stored in an object storage service module, then an importing task is generated, the importing task is pushed into a message queue, the task service module consumes the importing task in the message queue, and further the file to be imported in the object storage service module is stored in a database, and the database is a database of a service system, so that the file is imported into the service system.
In the export process, firstly, a data query condition is acquired, an export task is generated, the export task is pushed to a message queue, then the task service module consumes the export task in the message queue, further, structured data is queried from a database according to the data query condition, an attachment is acquired from the object storage service module, the structured data and the attachment form an export file, and then the export file is uploaded to the object storage service module to realize file export, so that the download of a user is facilitated.
Therefore, the embodiment of the invention manages the import task and the export task through the message queue by establishing the import task and the export task, and realizes the import and export operation based on the object storage service module. By the method, the file can be imported and exported asynchronously, after the user performs related operations at the front end, the rear end server sequentially executes the tasks in the message queue, so that the user can learn the importing result at the front end, obtain the downloading address and download the file, and the like, the user does not need to perform a large number of manual importing and exporting operations, the problems of slow importing and exporting speed of the multi-element data and overtime of the network are solved, the concurrency of long links of data processing is reduced, the stability of a service system is ensured, and the user experience is improved. Namely, the embodiment of the invention can reduce the waiting of users, prevent the overtime of the network, reduce the load of the server, improve the high availability of the application and also accord with the idea of micro-service.
In a second aspect, an embodiment of the present invention provides a data import and export device based on a message queue, including:
the first import module is used for uploading the file to be imported to the application service module so that the application service module can upload the file to be imported to the object storage service module for storage, an import task is generated, and the import task is pushed to the message queue; wherein the file to be exported comprises structured data and the attachment;
and the second import module is used for downloading a file to be imported corresponding to the import task from the object storage service module and importing the file to be imported into a database when the task service module monitors that a new task exists in the message queue and if the task is the import task.
In one embodiment, the apparatus further comprises:
and the first write-back module is used for writing back the state of the import task so as to ensure that a front-end user can check whether the import work of the file to be imported is successfully completed.
In one embodiment, the apparatus further comprises:
and the third import module is used for uploading the attachment to be imported into the object storage service module.
In one embodiment, the apparatus further comprises:
and the first storage module is used for storing the import task into the database.
In one embodiment, the apparatus further comprises:
the first receiving module is used for receiving the data query condition of the user;
the first pushing module is used for sending the data query condition to the application service module so that the application service module generates an export task and pushes the export task to a message queue;
and the first exporting module is used for querying corresponding structured data from the database according to the data query condition corresponding to the exporting task when the task service module monitors that the message queue has a new task, querying corresponding accessories from the object storage service module, generating a file to be exported from the structured data and the accessories, and uploading the file to be exported to the object storage service module so as to enable a user to download the file from the object storage service module at the front end.
In one embodiment, the apparatus further comprises:
and the second write-back module is used for writing back the state of the export task and the download address so as to ensure that a front-end user can check whether the export work of the export file is successfully completed or not, and downloads according to the download address when the export is successfully carried out.
In one embodiment, the apparatus further comprises:
and the second storage module is used for storing the export task to the database.
It may be understood that, for explanation, specific implementation, beneficial effects, examples, etc. of the content in the apparatus provided by the embodiment of the present invention, reference may be made to corresponding parts in the method provided in the first aspect, which are not repeated herein.
In a third aspect, embodiments of the present invention provide a computer readable medium having stored thereon computer instructions which, when executed by a processor, cause the processor to perform the method provided in the first aspect.
Specifically, a system or apparatus provided with a storage medium on which a software program code realizing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or apparatus may be caused to read out and execute the program code stored in the storage medium.
In this case, the program code itself read from the storage medium may realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code form part of the present invention.
Examples of the storage medium for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer by a communication network.
Further, it should be apparent that the functions of any of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform part or all of the actual operations based on the instructions of the program code.
Further, it is understood that the program code read out by the storage medium is written into a memory provided in an expansion board inserted into a computer or into a memory provided in an expansion module connected to the computer, and then a CPU or the like mounted on the expansion board or the expansion module is caused to perform part and all of actual operations based on instructions of the program code, thereby realizing the functions of any of the above embodiments.
It may be appreciated that, for explanation, specific implementation, beneficial effects, examples, etc. of the content in the computer readable medium provided by the embodiment of the present invention, reference may be made to corresponding parts in the method provided in the first aspect, and details are not repeated herein.
In a fourth aspect, one embodiment of the present specification provides a computing device comprising a memory having executable code stored therein and a processor that, when executing the executable code, performs the method of any one of the embodiments of the specification.
It may be appreciated that, for explanation, specific implementation, beneficial effects, examples, etc. of the content in the computing device provided by the embodiment of the present invention, reference may be made to corresponding parts in the method provided in the first aspect, which are not repeated herein.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.
Those skilled in the art will appreciate that in one or more of the examples described above, the functions described in the present invention may be implemented in hardware, software, a pendant, or any combination thereof. When implemented in software, these functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.
The foregoing embodiments have been provided for the purpose of illustrating the general principles of the present invention in further detail, and are not to be construed as limiting the scope of the invention, but are merely intended to cover any modifications, equivalents, improvements, etc. based on the teachings of the invention.

Claims (10)

1. A method for importing and exporting data based on a message queue, comprising:
uploading a file to be imported to an application service module, so that the application service module uploads the file to be imported to an object storage service module for storage, generates an import task, and pushes the import task to a message queue; wherein the file to be exported comprises structured data and the attachment;
and after the task service module monitors that a new task exists in the message queue, if the task is an import task, downloading a file to be imported corresponding to the import task from the object storage service module, and importing the file to be imported into a database.
2. The method as recited in claim 1, further comprising:
and writing back the state of the import task so as to enable a front-end user to check whether the import work of the file to be imported is successfully completed.
3. The method as recited in claim 1, further comprising:
and uploading the attachment to be imported into the object storage service module.
4. The method as recited in claim 1, further comprising:
and storing the import task into the database.
5. A method according to claim 3, further comprising:
receiving a data query condition of a user;
sending the data query condition to the application service module so that the application service module generates an export task and pushes the export task to a message queue;
and when the task service module monitors that the message queue has a new task, if the task is a exporting task, inquiring corresponding structured data from the database according to the data inquiry condition corresponding to the exporting task, inquiring corresponding accessories from the object storage service module, generating a file to be exported by the structured data and the accessories, and uploading the file to be exported to the object storage service module so as to enable a user to download the file from the object storage service module at the front end.
6. The method as recited in claim 5, further comprising:
and writing back the state of the export task and the download address so as to ensure that a front-end user checks whether the export work of the export file is successfully completed or not, and downloading according to the download address when the export work is successfully exported.
7. The method as recited in claim 5, further comprising:
and storing the export task to the database.
8. A message queue-based data import-export device, comprising:
the first import module is used for uploading the file to be imported to the application service module so that the application service module can upload the file to be imported to the object storage service module for storage, an import task is generated, and the import task is pushed to the message queue; wherein the file to be exported comprises structured data and the attachment;
and the second import module is used for downloading a file to be imported corresponding to the import task from the object storage service module and importing the file to be imported into a database when the task service module monitors that a new task exists in the message queue and if the task is the import task.
9. A computer readable storage medium, having stored thereon a computer program which, when executed in a computer, causes the computer to perform a method implementing any of claims 1 to 7.
10. A computing device comprising a memory and a processor, the memory having executable code stored therein, the processor, when executing the executable code, implementing the method of any one of claims 1-7.
CN202310364430.0A 2023-04-03 2023-04-03 Data import and export method and device based on message queue, medium and equipment Pending CN116431363A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310364430.0A CN116431363A (en) 2023-04-03 2023-04-03 Data import and export method and device based on message queue, medium and equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310364430.0A CN116431363A (en) 2023-04-03 2023-04-03 Data import and export method and device based on message queue, medium and equipment

Publications (1)

Publication Number Publication Date
CN116431363A true CN116431363A (en) 2023-07-14

Family

ID=87088531

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310364430.0A Pending CN116431363A (en) 2023-04-03 2023-04-03 Data import and export method and device based on message queue, medium and equipment

Country Status (1)

Country Link
CN (1) CN116431363A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116932252A (en) * 2023-09-18 2023-10-24 北京冠群信息技术股份有限公司 Asynchronous task compensation method and device based on batch data import pipeline

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116932252A (en) * 2023-09-18 2023-10-24 北京冠群信息技术股份有限公司 Asynchronous task compensation method and device based on batch data import pipeline
CN116932252B (en) * 2023-09-18 2024-01-26 北京冠群信息技术股份有限公司 Asynchronous task compensation method and device based on batch data import pipeline

Similar Documents

Publication Publication Date Title
CN101390080B (en) Serving cached query results based on a query portion
US8539107B2 (en) Personal information management data synchronization
US20030217197A1 (en) Preparation for software on demand system
CN110765206A (en) Data synchronization system, method, device, equipment and storage medium
CN116431363A (en) Data import and export method and device based on message queue, medium and equipment
CN110543324B (en) Plug-in increment updating method and device for application program
CN111159219B (en) Data management method, device, server and storage medium
CN102355487A (en) Browse handling method and device of email attachment
CN110659971B (en) Transaction data processing method and device
CN113051102B (en) File backup method, device, system, storage medium and computer equipment
CN113742392A (en) Data synchronization method and device, electronic equipment and storage medium
US8090769B2 (en) Dynamically generating web contents
CN111753141B (en) Data management method and related equipment
CN107896246B (en) Message propagation system based on XBRL
CN116108036A (en) Method and device for off-line exporting back-end system data
CN115658768A (en) Attribute query method and device for state database and related medium
CN107092529B (en) OLAP service method, device and system
US7895362B2 (en) Multiple message source electronic data interchange (EDI) enveloper with batching support
CN111405015B (en) Data processing method, device, equipment and storage medium
CN114063931A (en) Data storage method based on big data
CN114201659A (en) Message track transmission query method, device and system
CN113114557A (en) Message sending method and device, electronic equipment and storage medium
CN113407339A (en) Resource request feedback method and device, readable storage medium and electronic equipment
US20160080514A1 (en) Information repository system including a wireless device and related method
CN112527900A (en) Method, device, equipment and medium for database multi-copy reading consistency

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination