CN110737645B - Data migration method and system among different systems and related equipment - Google Patents
Data migration method and system among different systems and related equipment Download PDFInfo
- Publication number
- CN110737645B CN110737645B CN201910978951.9A CN201910978951A CN110737645B CN 110737645 B CN110737645 B CN 110737645B CN 201910978951 A CN201910978951 A CN 201910978951A CN 110737645 B CN110737645 B CN 110737645B
- Authority
- CN
- China
- Prior art keywords
- data
- page
- data packet
- target data
- packet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention provides a data migration method, a data migration system and related equipment among different systems, which are used for improving the efficiency of data migration. In the embodiment of the invention, the data migration system can acquire a data packet sent by a client of the export system to a server and browse page information returned by the server in the browsing operation process of target data in the client of the export system, analyzes system parameter information and browse page information in the data packet to obtain a page range of the target data, finally automatically simulates and generates a request data packet corresponding to each page of the target data by adopting an export crawler program according to the system parameter information, and acquires and stores data corresponding to each page of the target data according to the request data packet; and the data corresponding to each page number of the target data is stored in the import system by adopting an import crawler program, so that manual item-by-item import is not needed, and the efficiency of data migration is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data migration method between different systems, a data migration system, and a related device.
Background
With the rapid development of SAAS (software as a service) technology, more and more financial systems are on-line. For some reasons, both business and technology, customers need to migrate data between different financial systems.
In the existing data migration mode, a client needs to export data item by item manually and then import the data item by item into a new financial system again.
In the existing scheme, manual entry-by-entry leading-in and leading-out are adopted, so that the efficiency is low, and the integrity of data is easily influenced by human factors.
Disclosure of Invention
The embodiment of the invention provides a data migration method, a data migration system and related equipment among different systems, which are used for improving the efficiency of data migration.
A first aspect of an embodiment of the present invention provides a method for data migration between different systems, where the method includes:
receiving a data migration request, wherein the data migration request comprises login information in an export system;
logging in a client corresponding to an export system for storing the target data according to the login information, and acquiring a data packet sent to a server by the client of the export system in the process of browsing the target data in the client of the export system;
acquiring browsing page information returned by the server;
analyzing system parameter information in the data packet;
analyzing the browsing page information to obtain a page number range of the target data;
adopting a exporting crawler program to automatically simulate and generate a request data packet corresponding to each page number of the target data according to the system parameter information, and sending the request data packet to the server to acquire data corresponding to each page number of the target data;
and storing the data corresponding to each page number of the target data to an import system by adopting an import crawler program.
Optionally, as a possible implementation manner, in an embodiment of the present invention, the data packet follows a hypertext transfer HTTP protocol, and the analyzing the system parameter information in the data packet includes:
identifying a paging request packet in the data packets, wherein the paging request packet is used for requesting the server to browse corresponding content in a specified page number of the target data;
and analyzing the system parameter information in the paging request packet.
Optionally, as a possible implementation manner, in an embodiment of the present invention, the data packet follows an HTTP protocol, and the analyzing the system parameter information in the data packet includes:
identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated according to clicking of a preset position of a current page, and the operation instruction is used for indicating the server to return corresponding content in an appointed page number of the target data;
and analyzing the system parameter information in the operation instruction data packet.
Optionally, as a possible implementation manner, in the embodiment of the present invention, when data migration requests of multiple users are received, the data migration requests of the users are sequenced according to priorities or time sequences of the users.
Optionally, as a possible implementation manner, the method for migrating data between different systems in the embodiment of the present invention may further include:
acquiring classification information of the export system to all service data;
constructing a WEB page to display the classification information of the export system to all the service data;
and generating a data migration request according to the interaction operation of the WEB page.
Optionally, as a possible implementation manner, the method for migrating data between different systems in the embodiment of the present invention may further include:
and writing the data corresponding to each page number of the target data into a universal format file.
A second aspect of the present invention provides a data migration system, including:
the system comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a data migration request which comprises login information in an export system;
the acquisition unit is used for logging in a client corresponding to an export system for storing the target data according to the login information, and acquiring a data packet sent to a server by the client of the export system and browsing page information returned by the server in the browsing operation process of the target data in the client of the export system;
the first analysis unit is used for analyzing the system parameter information in the data packet;
the second analysis unit is used for analyzing the browsing page information to obtain the page number range of the target data;
the configuration unit is used for automatically simulating and generating a request data packet corresponding to each page of the target data according to the system parameter information by adopting a derived crawler program, and sending the request data packet to the server to acquire data corresponding to each page of the target data;
and the import unit is used for storing the data corresponding to each page number of the target data into an import system by adopting an import crawler program.
Optionally, as a possible implementation manner, in an embodiment of the present invention, the data packet follows an HTTP protocol, and the first parsing unit includes:
a first identification module, configured to identify a paging request packet in the data packet, where the paging request packet is used to request the server to browse corresponding content in a specified page number of the target data;
the first analysis module is used for analyzing the system parameter information in the paging request packet.
Optionally, as a possible implementation manner, in an embodiment of the present invention, the data packet conforms to an HTTP protocol, and the first parsing unit includes:
the second identification module is used for identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated according to clicking of a preset position of a current page, and the operation instruction is used for indicating the server to return corresponding content in an appointed page number of the target data;
and the second analysis module is used for analyzing the system parameter information in the operation instruction data packet.
A third aspect of embodiments of the present application provides a computer apparatus, where the computer apparatus includes a processor, and the processor is configured to implement the steps in any one of the first aspect and the possible implementation manner of the first aspect when executing a computer program stored in a memory.
A fourth aspect of embodiments of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein: the computer program realizes the steps of any of the possible implementations of the first aspect and the first aspect when executed by a processor.
According to the technical scheme, the embodiment of the invention has the following advantages:
in the embodiment of the invention, a data migration system can log in a client corresponding to a export system for storing target data, acquire a data packet sent by the client of the export system to a server and browsing page information returned by the server in the browsing operation process of the target data in the client of the export system, analyze system parameter information and the browsing page information in the data packet to obtain a page range of the target data, finally automatically simulate and generate a request data packet corresponding to each page of the target data by adopting an export crawler according to the system parameter information, and acquire and store data corresponding to each page of the target data according to the request data packet; and the data corresponding to each page number of the target data is stored in the import system by adopting an import crawler program, so that manual item-by-item import is not needed, and the efficiency of data migration is improved.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a method for data migration between different systems according to an embodiment of the present invention;
FIG. 2 is a timing diagram illustrating an exemplary implementation of a method for data migration between different systems according to an embodiment of the present invention;
FIG. 3 is a diagram of an embodiment of a data migration system in accordance with an embodiment of the present invention;
FIG. 4 is a diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a data migration method, a data migration system and related equipment among different systems, which are used for improving the efficiency of data migration.
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
For convenience of understanding, a detailed flow in the embodiment of the present invention is described below, and referring to fig. 1, an embodiment of a method for migrating data between different systems in the embodiment of the present invention may include:
101. receiving a data migration request;
when a user needs to migrate data between different systems, a data migration request can be submitted according to an interactive interface provided by the data migration system, wherein the data migration request comprises login information in the export system. The export system is a storage system for currently storing selected target data, for example, a financial system for storing account set data (basic information, subjects, balances, certificates, auxiliary accounting, fixed assets, wages, and the like), and the import system and the export system are two independent and different storage systems. The login information may be a combination of a user name and a password, or may be other data information for verifying the identity of the user, such as fingerprint data, facial feature data, and the like, which is not limited herein.
It is understood that one data migration request may include one or more types of target data fields to complete the migration of multiple types of target data, for example, a financial system may simultaneously select to migrate all types of data in one account set, and the specific amount of target data is not limited herein.
102. Logging in a client corresponding to an export system for storing target data according to the login information, and acquiring a data packet sent to a server by the client of the export system and browsing page information returned by the server in the process of browsing the target data in the client of the export system;
in the embodiment of the present invention, the applicant notices that the server transmits the data stored in the server to the client during the data browsing operation in the export system. In view of this, in order to acquire the target data stored in the export system, after acquiring the data migration request, the client corresponding to the export system storing the target data may be logged in according to the login information, and the client of the export system may be queried for the target data according to the type field of the target data, so as to perform a browsing operation on the target data, so that the server sends the target data to the local client. The client of the export system may be a WEB page client implemented based on a browser, or may be an individual APP client, and the specific type of the client is not limited here.
During the browsing operation process of the target data, the client of the export system interacts with the corresponding server, and sends a data packet to the server during the interaction process, the data migration system can acquire the data packet sent by the local client and browsing page information returned by the server, and the browsing page information is used for constructing a browsing page at the client for a user to browse and view.
It is understood that the data browsing process may be a manual operation or an automatic operation of the data migration system, for example, an interactive page may be provided, where the interactive page includes fields of the export system and the import system selected by the user, the field of the type of the target data selected by the user is written into the data migration request, and the data browsing operation is automatically initiated according to the field of the type of the target data selected by the user.
103. Analyzing system parameter information in the data packet;
after the data packet sent by the local client is obtained, the data migration system may analyze system parameter information in the data packet, so as to subsequently simulate the data packet to perform automatic data extraction, specifically, the system parameter information is used to instruct the server to return browsing page information corresponding to browsing operation of the target data, and may include a system parameter type and a parameter configuration format, and the specific system parameter information is set by the export system, which is not limited herein.
In practical applications, when a client needs to obtain target data, the client may send two types of data packets to a server to achieve the purpose. One is a paging request packet and the other is an operation instruction packet. The paging request data packet is used for requesting a server to browse corresponding content in a specified page of target data, and often includes a Uniform Resource Locator (URL) address for storing the target data and an identification field of the specified page. The other is an operation instruction data packet, where the operation instruction data packet is an operation instruction generated according to a click of a preset position of a current page, and the operation instruction is used to instruct the server to return corresponding content in a specified page number of target data, for example, when a document is browsed, a "next page" option exists in the preset position of a currently displayed page of the client, and when the option is clicked, the client generates an operation instruction data packet and sends the operation instruction data packet to the server.
Optionally, to facilitate data parsing, the data packet needs to conform to a public transport protocol, such as a hypertext transport protocol HTTP protocol, and as a possible real-time manner, parsing the system parameter information in the data packet includes: identifying a paging request packet in the data packet, wherein the paging request packet is used for requesting a server to browse corresponding contents in specified page numbers of target data; and analyzing the system parameter information in the paging request packet.
Optionally, the data packet follows an HTTP protocol, and as a possible real-time manner, analyzing the system parameter information in the data packet includes: identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated by clicking a preset position of a current page, and the operation instruction is used for indicating a server to return corresponding content in an appointed page number of target data; and analyzing the system parameter information in the operation instruction data packet.
104. Analyzing the browsing page information to obtain the page number range of the target data;
in the process of browsing data, target data is often browsed page by page, browsed page information returned by the server also comprises an overall page number range identifier and a current page number identifier, and the data migration system can analyze the browsed page information to obtain the page number range of the target data.
105. Adopting a exporting crawler program to automatically simulate and generate a request data packet corresponding to each page of the target data according to the system parameter information, and sending the request data packet to a client-side server of the exporting system so as to acquire and store data corresponding to each page of the target data;
in order to improve the efficiency of data export, an export crawler program can be called to automatically simulate and generate a request data packet corresponding to each page of the target data according to the system parameter information, and the request data packet is sent to a client-side server of the export system so as to acquire and store data corresponding to each page of the target data.
Specifically, when the paging request packet is adopted, data corresponding to each page of the target data can be obtained only by calling a crawler program to call an interface corresponding to the server and sequentially sending the request data packet corresponding to each page of the target data. When the operation instruction data packet is adopted, after the crawler program is required to be called to return the data corresponding to the previous page number, the operation instruction data packet is sent to the server and used for indicating the server to return the data in the next page number of the current page number until the last page number.
106. And storing the data corresponding to each page number of the target data into an import system by adopting an import crawler program.
The data migration system can call an import crawler program to store the data corresponding to each page number of the target data into the import system. Specifically, the data can be automatically stored by logging in the import system according to the login information of the import system input by the user. In practical application, format conversion can be performed according to format requirements of different storage systems, and particularly, reasonable setting can be performed according to requirements, which is not limited here.
It should be understood that, in various embodiments of the present invention, the sequence numbers of the above steps do not mean the execution sequence, and the execution sequence of each step should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. For example, the order of execution of step 104 and step 103 may be interchanged.
In the embodiment of the invention, a data migration system can log in a client corresponding to a export system for storing target data, acquire a data packet sent by the client of the export system to a server and browsing page information returned by the server in the browsing operation process of the target data in the client of the export system, analyze system parameter information and the browsing page information in the data packet to obtain a page range of the target data, finally automatically simulate and generate a request data packet corresponding to each page of the target data by adopting an export crawler according to the system parameter information, and acquire and store data corresponding to each page of the target data according to the request data packet; and the data corresponding to each page number of the target data is stored in the import system by adopting an import crawler program, so that manual item-by-item import is not needed, and the efficiency of data migration is improved.
Optionally, on the basis of the embodiment shown in fig. 1, in order to improve the interaction efficiency with the user, as a possible implementation manner, the data migration system may pre-construct a WEB page for interaction, which specifically includes: acquiring classification information of an export system on all service data; constructing classification information of a WEB page display export system on all service data; and generating a data migration request according to the interaction operation of the WEB page.
For example, taking a financial system as an example, the data migration system may call a crawler program to log in the financial system according to login information provided by a user, construct a WEB page to display account set data and an optional import system in the financial system according to account set data (account set basic information, subjects, balances, certificates, auxiliary accounting, fixed assets, salaries, and the like) in the financial system, and generate a data migration request according to a click operation of the user on the WEB page.
Optionally, on the basis of the embodiment shown in fig. 1, in order to improve the execution efficiency of the data transfer tasks, it is necessary to manage the order of executing the multiple data transfer tasks, and as a possible implementation manner, when receiving data transfer requests of multiple users, the data transfer requests of the users are sorted according to the priority or time order of the users. For example, the level of the account may be set, and the higher the level of the account, the earlier the execution sequence of the data transfer task corresponding to the account is, the accounts in the same level are sorted according to the time of task submission.
Optionally, on the basis of the embodiment shown in fig. 1, when the import is not supported temporarily during the maintenance of the import system, the data corresponding to each page number of the target data may be written into a common format file (e.g., an XML format file), and a re-import operation is performed after the recovery of the import system.
For convenience of understanding, the following description will be made on different inter-system data migration methods in the embodiments of the present invention with reference to specific application examples. Referring to the timing diagram of fig. 2, the data migration system may include a WEB interface interactive program, a exporting crawler, and an importing crawler. Both the export website and the import website support the HTTP transmission protocol, and for example, basic information, subjects, balance, certificates, auxiliary accounting, fixed assets, wages, and the like of the account set are exported.
Because online SAAS (software as a service) systems are generally accessed using HTTP/HTTPs protocols, which are a public protocol, there are many tools that can perform detailed protocol analysis at present, so that details of HTTP interfaces and data packets can be known without depending on documents of the original system.
The specific implementation process is as follows: the user selects imported/exported products according to interfaces displayed by the provided WEB interface interactive program, the WEB interface interactive program calls a crawler to acquire login elements of an exported website and an imported website, and login pages of the exported website and the imported website are constructed and displayed according to login interface setting parameters returned by the exported website and the imported website. After a user inputs login information (such as a user name and a password) of a exporting website and a importing website on a login page, a crawler is exported and calls a login of the exporting website and an interface for acquiring an account set list to acquire the account set list to a WEB interface interactive program, and the WEB interface interactive program displays the optional account set list on the interactive page. After a user selects account set data needing to be exported, a crawler for exporting financial data is compiled by analyzing system parameter information of HTTP request data packets of all business requests of an exporting website, then system parameter information of the HTTP request data packets of all business requests of the importing website is analyzed, a WEB interface interaction program of the importing crawler for compiling financial data can call the exporting crawler to automatically send the simulated HTTP request data packets to the exporting website and the importing website so as to automatically acquire data such as account set basic information, subjects, balance, certificates, auxiliary accounting, fixed assets, wages and the like returned by the exporting website without clicking operation item by the user. Optionally, the WEB interface interactive program may store data such as the basic information of the account set, the subject, the balance, the certificate, the auxiliary accounting, the fixed asset, the wage, and the like returned by the export website to a file of a general format, for example, a standard XML file, or may directly call the import crawler program to store the data to the import website without intermediate transfer.
In the embodiment of the invention, the data of the SAAS financial system is decomposed into basic information, subjects, certificates, auxiliary accounting items, balance, fixed assets, wages and the like, and then the basic information, the subjects, the certificates, the auxiliary accounting items, the balance, the fixed assets, the wages and the like are defined by a standard XML format and can be used for other products or the public, so that the exporting part and the importing part can be separated, the resources of everyone are centralized to quickly finish the requirements of customers, and the system has longer vitality. Secondly, analyzing and clearly exporting/importing each data access interface of the website through analyzing the HTTP protocol of the website, exporting/importing and storing the data by using a crawler program, and improving the efficiency of data migration between different systems.
Referring to fig. 3, an embodiment of the present invention further provides a data migration system, which may include:
a receiving unit 301, configured to receive a data migration request, where the data migration request includes login information in an export system;
an obtaining unit 302, configured to log in a client corresponding to an export system storing target data according to login information, and obtain a data packet sent by the client of the export system to a server and browsing page information returned by the server in a process of browsing the target data in the client of the export system;
a first parsing unit 303, configured to parse system parameter information in the data packet;
a second parsing unit 304, configured to parse the browsing page information to obtain a page range of the target data;
the configuration unit 305 is configured to generate a request data packet corresponding to each page of the target data by using a derived crawler program according to the system parameter information through automatic simulation, and send the request data packet to the server to obtain data corresponding to each page of the target data;
and the import unit 306 is configured to store data corresponding to each page of the target data in an import system by using an import crawler program.
In the embodiment of the invention, a data migration system can log in a client corresponding to a export system for storing target data, acquire a data packet sent by the client of the export system to a server and browsing page information returned by the server in the browsing operation process of the target data in the client of the export system, analyze system parameter information and the browsing page information in the data packet to obtain a page range of the target data, finally automatically simulate and generate a request data packet corresponding to each page of the target data by adopting an export crawler according to the system parameter information, and acquire and store data corresponding to each page of the target data according to the request data packet; and the data corresponding to each page number of the target data is stored in the import system by adopting an import crawler program, so that manual item-by-item import is not needed, and the efficiency of data migration is improved.
Optionally, as a possible implementation manner, the data packet in the embodiment of the present invention follows a hypertext transfer protocol HTTP protocol, and the first parsing unit includes:
the first identification module is used for identifying a paging request packet in the data packet, wherein the paging request packet is used for requesting a server to browse corresponding contents in specified page numbers of target data;
the first analysis module is used for analyzing the system parameter information in the paging request packet.
Optionally, as a possible implementation manner, the data packet in the embodiment of the present invention follows an HTTP protocol, and the first parsing unit includes:
the second identification module is used for identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated according to clicking of a preset position of a current page, and the operation instruction is used for indicating the server to return corresponding content in a specified page number of target data;
and the second analysis module is used for analyzing the system parameter information in the operation instruction data packet.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
Fig. 4 is a schematic diagram illustrating only a portion related to an embodiment of the present application for convenience of description, and please refer to a method portion of the embodiment of the present application for details not disclosed. Referring to fig. 4, the computer apparatus 4 includes: memory 410, processor 420, wired or wireless network module 430, and computer programs stored in the memory and executable on the processor. The processor, when executing the computer program, implements the steps of the various embodiments of the inter-system data migration method described above, such as steps 101 through 106 shown in fig. 1. Alternatively, the processor, when executing the computer program, implements the functions of each module or unit in the above-described apparatus embodiments.
In some of the embodiments of the application, the processor, when executing the computer program stored in the memory, implements the steps of:
receiving a data migration request, wherein the data migration request comprises login information in an export system;
logging in a client corresponding to an export system for storing target data according to the login information, and acquiring a data packet sent to a server by the client of the export system in the process of browsing the target data in the client of the export system;
acquiring browsing page information returned by a server;
analyzing system parameter information in the data packet;
analyzing the browsing page information to obtain a page number range of the target data;
adopting a exporting crawler program to automatically simulate and generate a request data packet corresponding to each page of the target data according to the system parameter information, and sending the request data packet to a server to obtain data corresponding to each page of the target data;
and storing the data corresponding to each page number of the target data into an import system by adopting an import crawler program.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
identifying a paging request packet in the data packet, wherein the paging request packet is used for requesting a server to browse corresponding contents in the designated page number of the target data;
and analyzing the system parameter information in the paging request packet.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated by clicking a preset position of a current page, and the operation instruction is used for indicating a server to return corresponding content in an appointed page number of target data;
and analyzing the system parameter information in the operation instruction data packet.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
when data migration requests of a plurality of users are received, the data migration requests of the users are sequenced according to the priorities or time sequence of the users.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
acquiring classification information of all service data of an export system;
constructing classification information of a WEB page display export system on all service data;
and generating a data migration request according to the interaction operation of the WEB page.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
and writing data corresponding to each page number of the target data into the universal format file.
Those skilled in the art will appreciate that the configuration shown in fig. 4 does not constitute a limitation of the computer apparatus, and that computer apparatus 4 may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components, e.g., the computer apparatus may also include input-output devices, buses, etc.
The Processor may be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory may be used to store computer programs and/or modules, and the processor may implement various functions of the computer device by executing or executing the computer programs and/or modules stored in the memory, as well as by invoking data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; the storage data area may store data (such as audio data, a phonebook, etc.) created according to the use of the cellular phone, etc. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The present application further provides a computer-readable storage medium having a computer program stored thereon, which when executed by a processor, can implement the steps of:
receiving a data migration request, wherein the data migration request comprises login information in an export system;
logging in a client corresponding to an export system for storing target data according to the login information, and acquiring a data packet sent to a server by the client of the export system in the process of browsing the target data in the client of the export system;
acquiring browsing page information returned by a server;
analyzing system parameter information in the data packet;
analyzing the browsing page information to obtain the page number range of the target data;
adopting a exporting crawler program to automatically simulate and generate a request data packet corresponding to each page of the target data according to the system parameter information, and sending the request data packet to a server to acquire data corresponding to each page of the target data;
and storing the data corresponding to each page number of the target data into an import system by adopting an import crawler program.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
identifying a paging request packet in the data packet, wherein the paging request packet is used for requesting a server to browse corresponding contents in the designated page number of the target data;
and analyzing the system parameter information in the paging request packet.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated by clicking a preset position of a current page, and the operation instruction is used for indicating a server to return corresponding content in an appointed page number of target data;
and analyzing the system parameter information in the operation instruction data packet.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
when data migration requests of a plurality of users are received, the data migration requests of the users are sequenced according to the priorities or time sequence of the users.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
acquiring classification information of all service data of an export system;
constructing classification information of a WEB page display export system on all service data;
and generating a data migration request according to the interaction operation of the WEB page.
Optionally, in some embodiments of the present application, the processor may be further configured to implement the following steps:
and writing the data corresponding to each page number of the target data into the universal format file.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (7)
1. A method for data migration between different systems, comprising:
receiving a data migration request, wherein the data migration request comprises login information in an export system;
logging in a client corresponding to an export system for storing target data according to the login information, and acquiring a data packet sent to a server by the client of the export system in the process of browsing the target data in the client of the export system;
acquiring browsing page information returned by the server;
analyzing system parameter information in the data packet;
analyzing the browsing page information to obtain a page number range of the target data;
adopting a exporting crawler program to automatically simulate and generate a request data packet corresponding to each page of the target data according to the system parameter information, and sending the request data packet to the server to obtain data corresponding to each page of the target data;
adopting an import crawler program to store data corresponding to each page number of the target data into an import system;
wherein the data packets conform to the hypertext transfer protocol (HTTP);
the analyzing the system parameter information in the data packet comprises:
identifying a paging request packet in the data packet, wherein the paging request packet is used for requesting the server to browse corresponding contents in the specified page number of the target data;
analyzing system parameter information in the paging request packet;
or, the analyzing the system parameter information in the data packet includes:
identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated according to clicking of a preset position of a current page, and the operation instruction is used for indicating the server to return corresponding content in an appointed page number of the target data;
and analyzing the system parameter information in the operation instruction data packet.
2. The method according to claim 1, wherein when data migration requests of a plurality of users are received, the data migration requests of the users are sorted according to the priority or time sequence of the users.
3. The method of claim 1, further comprising:
acquiring classification information of the export system to all service data;
constructing a WEB page to display the classification information of the export system to all the service data;
and generating a data migration request according to the interaction operation of the WEB page.
4. The method of claim 1, further comprising:
and writing the data corresponding to each page number of the target data into a universal format file.
5. A data migration system, comprising:
the system comprises a receiving unit, a sending unit and a receiving unit, wherein the receiving unit is used for receiving a data migration request which comprises login information in an export system;
the acquisition unit is used for logging in a client corresponding to an export system for storing target data according to the login information, and acquiring a data packet sent to a server by the client of the export system and browsing page information returned by the server in the process of browsing the target data in the client of the export system;
the first analysis unit is used for analyzing the system parameter information in the data packet;
the second analysis unit is used for analyzing the browsing page information to obtain the page number range of the target data;
the configuration unit is used for automatically simulating and generating a request data packet corresponding to each page of the target data according to the system parameter information by adopting a derived crawler program, and sending the request data packet to the server to acquire data corresponding to each page of the target data;
the import unit is used for storing the data corresponding to each page number of the target data into an import system by adopting an import crawler program;
wherein the data packet conforms to the HTTP protocol;
the first parsing unit includes:
a first identification module, configured to identify a paging request packet in the data packet, where the paging request packet is used to request the server to browse corresponding content in a specified page number of the target data;
the first analysis module is used for analyzing the system parameter information in the paging request packet;
alternatively, the first parsing unit includes:
the second identification module is used for identifying an operation instruction data packet in the data packet, wherein the operation instruction data packet is an operation instruction generated according to clicking of a preset position of a current page, and the operation instruction is used for indicating the server to return corresponding content in an appointed page number of the target data;
and the second analysis module is used for analyzing the system parameter information in the operation instruction data packet.
6. A computer arrangement, characterized in that the computer arrangement comprises a processor for implementing the steps of the method according to any one of claims 1-4 when executing a computer program stored in a memory.
7. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implementing the steps of the method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910978951.9A CN110737645B (en) | 2019-10-15 | 2019-10-15 | Data migration method and system among different systems and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910978951.9A CN110737645B (en) | 2019-10-15 | 2019-10-15 | Data migration method and system among different systems and related equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737645A CN110737645A (en) | 2020-01-31 |
| CN110737645B true CN110737645B (en) | 2022-09-27 |
Family
ID=69268901
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910978951.9A Active CN110737645B (en) | 2019-10-15 | 2019-10-15 | Data migration method and system among different systems and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737645B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609427B (en) * | 2021-08-06 | 2023-09-08 | 山东鸿业信息科技有限公司 | System data resource extraction method and system under no-interface condition |
| CN114640974B (en) * | 2022-05-16 | 2022-10-21 | 荣耀终端有限公司 | Data migration method, device and storage medium |
| CN117271482B (en) * | 2023-11-23 | 2024-02-27 | 成方金融信息技术服务有限公司 | Historical data migration method and device, electronic equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662766A (en) * | 2012-03-23 | 2012-09-12 | 北京邮电大学 | Microblog and social networking information synchronizing system and synchronizing method based on Web Element mechanism |
| CN105224420A (en) * | 2015-10-08 | 2016-01-06 | 惠州Tcl移动通信有限公司 | A kind of analytical approach of automatic parsing terminal abnormal and system |
| US9736246B1 (en) * | 2015-02-19 | 2017-08-15 | Amazon Technologies, Inc. | Cross-device synchronization system for account-level information |
| CN107734019A (en) * | 2017-09-29 | 2018-02-23 | 中国石油化工股份有限公司 | The system and method that exploration and development container migrates online |
| CN108090091A (en) * | 2016-11-23 | 2018-05-29 | 北京国双科技有限公司 | Web page crawl method and apparatus |
| CN108667809A (en) * | 2018-04-13 | 2018-10-16 | 三维通信股份有限公司 | A kind of implementation method of multisystem DAS WEB interfaces account information management by synchronization |
| CN109428877A (en) * | 2017-09-01 | 2019-03-05 | 百度在线网络技术（北京）有限公司 | A kind of method and apparatus for by user equipment access operation system |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104104653B (en) * | 2013-04-03 | 2018-02-23 | 中国移动通信集团公司 | A kind of synchronous method, apparatus and system of page info |
| CN103927384A (en) * | 2014-04-30 | 2014-07-16 | 国家电网公司 | Multi-hierarchy data synchronism method and system between heterogeneous systems through HTML script analysis |
| CN103955517B (en) * | 2014-05-05 | 2017-05-03 | 中国工商银行股份有限公司 | Method and system for converting data in documental database to relational database |
-
2019
- 2019-10-15 CN CN201910978951.9A patent/CN110737645B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102662766A (en) * | 2012-03-23 | 2012-09-12 | 北京邮电大学 | Microblog and social networking information synchronizing system and synchronizing method based on Web Element mechanism |
| US9736246B1 (en) * | 2015-02-19 | 2017-08-15 | Amazon Technologies, Inc. | Cross-device synchronization system for account-level information |
| CN105224420A (en) * | 2015-10-08 | 2016-01-06 | 惠州Tcl移动通信有限公司 | A kind of analytical approach of automatic parsing terminal abnormal and system |
| CN108090091A (en) * | 2016-11-23 | 2018-05-29 | 北京国双科技有限公司 | Web page crawl method and apparatus |
| CN109428877A (en) * | 2017-09-01 | 2019-03-05 | 百度在线网络技术（北京）有限公司 | A kind of method and apparatus for by user equipment access operation system |
| CN107734019A (en) * | 2017-09-29 | 2018-02-23 | 中国石油化工股份有限公司 | The system and method that exploration and development container migrates online |
| CN108667809A (en) * | 2018-04-13 | 2018-10-16 | 三维通信股份有限公司 | A kind of implementation method of multisystem DAS WEB interfaces account information management by synchronization |
Non-Patent Citations (1)
| Title |
|---|
| 一种通用远程数据迁移汇总方案与实现;郭晓云;《电脑编程技巧与维护》;20150228(第2期);47-50 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737645A (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107402880B (en) | Test method and electronic equipment | |
| CN110737645B (en) | Data migration method and system among different systems and related equipment | |
| US20130136253A1 (en) | System and method for tracking web interactions with real time analytics | |
| US20160173486A1 (en) | Method and system for automating submission of issue reports | |
| CN108334641B (en) | Method, system, electronic equipment and storage medium for collecting user behavior data | |
| CN110798445B (en) | Public gateway interface testing method and device, computer equipment and storage medium | |
| CN111176963B (en) | Service evaluation information processing method and device | |
| US20160012074A1 (en) | System and method for providing contextual analytics data | |
| TWI791176B (en) | Method, system, device and computer program carrier for automatically identifying effective data collection modules | |
| CN110263070A (en) | Event report method and device | |
| CN104301148A (en) | User behavior recording method based on website access | |
| CN103399968B (en) | A kind of micro-blog information acquisition method and system | |
| CN111953766A (en) | Method and system for collecting network data | |
| CN109862074B (en) | Data acquisition method and device, readable medium and electronic equipment | |
| CN105184559B (en) | A kind of payment system and method | |
| CN107368407A (en) | Information processing method and device | |
| CN106202357A (en) | A kind of website browsing data analysing method and device | |
| CN108011936A (en) | Method and apparatus for pushed information | |
| CN108345508A (en) | Interface calls test method and device | |
| CN105335470B (en) | User login information shows method and apparatus | |
| CN111124858A (en) | Embedded point-free plug-in data processing method and system, terminal device and storage medium | |
| CN107798085B (en) | Service processing method and device based on plug-in | |
| CN112749975B (en) | Method for automatically processing refund request and automatic processing platform | |
| CN115640325A (en) | Content recommendation method and device, readable medium and electronic equipment | |
| CN113672497A (en) | Method, device and equipment for generating non-buried point event and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |