CN113641579A

CN113641579A - Data processing method, device, equipment and storage medium

Info

Publication number: CN113641579A
Application number: CN202110915623.1A
Authority: CN
Inventors: 谢翠云
Original assignee: Ping An Life Insurance Company of China Ltd
Current assignee: Ping An Life Insurance Company of China Ltd
Priority date: 2021-08-10
Filing date: 2021-08-10
Publication date: 2021-11-12

Abstract

The invention provides a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: and receiving a data acquisition request of the first system for the specified project through the mock platform, analyzing the data acquisition request, acquiring domain name information of a second system where the required target data is located and URI (Uniform resource identifier) matching rules of the first system and the second system, acquiring the target data, and sending the acquired target data to the first system. The invention has the beneficial effects that: the first system and the second system can not generate direct interaction of data, and the problem of increased coupling between the first system and the second system is reduced.

Description

Data processing method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data processing, and in particular, to a data processing method, apparatus, device, and storage medium.

Background

With the development and expansion of software projects, the conversion from single application to micro-service is experienced, but the problem of mutual influence among systems cannot be avoided no matter the single application or the micro-service application is used, particularly in the test process, data interaction is frequent, the coupling among the systems is easy to deepen, and therefore the performance of the systems is reduced. Therefore, a need exists for a system that reduces the increased coupling problem of the system during data interaction.

Disclosure of Invention

The invention mainly aims to provide a data processing method, a data processing device, data processing equipment and a data processing storage medium, and aims to solve the problem that when data interaction is frequent, coupling between systems is easily deepened, and therefore performance of the systems is reduced.

The invention provides a data processing method, which comprises the following steps:

the mock platform receives a data acquisition request of a first system for a specified project;

analyzing the data acquisition request, and acquiring URI information of target data required in the data acquisition request;

acquiring domain name information of a second system and a URI matching rule of the second system based on the URI information;

sending a request for acquiring the target data to the second system based on the domain name information and the URI matching rule;

and when target data sent by the second system is received, forwarding the target data to the first system.

Further, mock service data is preset in the mock platform, and before the step of obtaining the domain name information of the second system and the URI matching rule of the second system based on the URI information, the method further includes:

judging whether the mock service data contains the target data or not;

and if the target data is contained, directly sending the target data in the mock data to the first system.

Further, after the step of sending a request for obtaining the target data to the second system based on the domain name information and the URI matching rule, the method further includes:

if the target data is not returned by the second system or the returned data is unavailable, detecting whether the data acquisition request has a confirmation instruction for receiving the data in the mock platform;

if the confirmation instruction exists, searching for alternative data corresponding to the target data in the mock platform based on the URI information;

sending the replacement data to the first system.

Further, before the step of receiving a data acquisition request of the first system for a specified item, the mock platform further includes:

acquiring and counting the frequency information of acquiring various data by the first system in a specified time period through an sqoop script;

establishing a data list which needs to be acquired in advance in various data according to the acquired frequency information;

and acquiring and storing corresponding data from a corresponding system based on the data list.

Further, after the step of obtaining and storing the corresponding data from the corresponding system based on the data list, the method further includes:

and if the actual use data of the first system to the specified project is received, updating the corresponding data stored in the platform by the actual use data, and forwarding an update notification to the corresponding system.

Further, after the step of sending the request for obtaining the target data to the second system based on the domain name information and the URI matching rule, the method further includes:

receiving a test request for the specified item from a front end;

splicing the acquired target data and the data acquisition request into a Json file;

and sending the Json file to the front end, and testing the specified project by the front end.

Further, the step of forwarding the target data to the first system includes:

vectorizing the target data and the data acquisition request to respectively obtain a corresponding data vector and a corresponding request vector;

inputting the data vector and the request vector into a preset decoupling formula to obtain a decoupled data vector;

and sending the data corresponding to the decoupled data vector to the first system.

The present invention also provides a data processing apparatus, comprising:

the receiving module is used for receiving a data acquisition request of a first system for a specified project;

the analysis module is used for analyzing the data acquisition request and acquiring URI information of target data required in the data acquisition request;

the acquisition module is used for acquiring domain name information of a second system and a URI matching rule of the second system based on the URI information;

a sending module, configured to send a request for obtaining the target data to the second system based on the domain name information and the URI matching rule;

and the forwarding module is used for forwarding the target data to the first system when receiving the target data sent by the second system.

The invention also provides a computer device comprising a memory storing a computer program and a processor implementing the steps of any of the above methods when the processor executes the computer program.

The invention also provides a computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the method of any of the above.

The invention has the beneficial effects that: and receiving a data acquisition request of the first system for the specified project through the mock platform, analyzing the data acquisition request, acquiring domain name information of a second system where the required target data is located and URI (Uniform resource identifier) matching rules of the first system and the second system, acquiring the target data, and sending the acquired target data to the first system. Therefore, direct interaction of data between the first system and the second system cannot occur, and the problem of increased coupling between the first system and the second system is reduced.

Drawings

FIG. 1 is a flow chart of a data processing method according to an embodiment of the invention;

FIG. 2 is a block diagram schematically illustrating a data processing apparatus according to an embodiment of the present invention;

fig. 3 is a block diagram illustrating a structure of a computer device according to an embodiment of the present application.

The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship between the components, the motion situation, etc. in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indicator is changed accordingly, and the connection may be a direct connection or an indirect connection.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and B, may mean: a exists alone, A and B exist simultaneously, and B exists alone.

In addition, the descriptions related to "first", "second", etc. in the present invention are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.

Referring to fig. 1, the present invention provides a data processing method, including:

s1: the mock platform receives a data acquisition request of a first system for a specified project;

s2: analyzing the data acquisition request, and acquiring URI information of target data required in the data acquisition request;

s3: acquiring domain name information of a second system and URI matching rules of the first system and the second system based on the URI information;

s4: sending a request for acquiring the target data to the second system based on the domain name information and the URI matching rule;

s5: and when target data sent by the second system is received, forwarding the target data to the first system.

As described in the above step S1, the mock platform receives a data acquisition request of the first system for a specified item. The mock platform is used as a test service technology for simulating data, is equivalent to a hub for data interaction between systems, can store corresponding data inside, can be manually compiled and uploaded, can also be uploaded from other places in advance, and can interact with a plurality of systems. Therefore, according to the data acquisition request uploaded by the first system, corresponding data can be acquired from other systems and uploaded to the systems. The designated item is generally an item uploaded to the mock platform by related personnel, such as a test item, and the data acquisition request of the designated item includes information of target data required by the test item.

As described in step S2, the data acquisition request is analyzed to acquire URI information of target data required in the data acquisition request. The URI information (Uniform Resource Identifier) is the location information of the target data, and the data information of the target data, such as the format and size of the target data information.

As described in step S3 above, the domain name information of the second system and the URI matching rule with the second system are obtained based on the URI information. The second system is a system where the target data is located, the URI matching rule is an interface for acquiring the target data, and an interface protocol, and any one of the platform and the system can acquire the related data from the second system based on the URI matching rule. Therefore, the mock platform can send a request for obtaining the target data to the second system based on the URI matching rule, and the second system sends the target data to the mock platform in response to the request. The first system and the second system may be test systems, the first system generally being an upstream system and the second system generally being a downstream system. Set up URI matching rule in the mock platform can save the time that the mock platform obtained URI matching rule, otherwise still need follow other systems, or first system obtains, need spend extra time, this application directly is provided with corresponding URI matching rule in the mock platform, can realize faster acquisition to the target data.

As described in step S4 above, a request for obtaining the target data is sent to the second system based on the domain name information and the URI matching rule. I.e. upon sending a data acquisition request to the second system, which includes the interface required for the target data transfer, and the url format of the interface. Thereby obtaining the target data.

As described in step S5 above, it is determined whether the target data transmitted by the second system is available. In addition, the data is processed in a mock platform mode, and the target data is forwarded, so that the invasion to the first system and the second system is reduced, the coupling between the first system and the second system is further reduced, and the overall performance of the first system and the second system is improved. In addition, before sending, the target data can be decoupled, and the coupling relation between the target data and the designated item is reduced, so that the coupling between the first system and the second system is reduced.

Further, before step S3 of obtaining domain name information of the second system and a URI matching rule with the second system based on the URI information, the mock platform pre-sets mock service data therein, and further includes:

s201: judging whether the mock service data contains the target data or not;

s202: and if the target data is contained, directly sending the target data in the mock data to the first system.

As described in the foregoing steps S201 to S202, fast forwarding of the target data is implemented, in some embodiments, in order to facilitate fast response to the data acquisition request of the first system, the mock system may acquire some data from the second system in advance, and if the acquired data exactly includes the target data, the target data may be directly sent to the first system, so that fast forwarding of the target data is implemented, and the efficiency of the first system in acquiring data is improved.

In one embodiment, after the step S4 of sending the request for obtaining the target data to the second system based on the domain name information and the URI matching rule, the method further includes:

s501: if the target data is not returned by the second system or the returned data is unavailable, detecting whether the data acquisition request has a confirmation instruction for receiving the data in the mock platform;

s502: if the confirmation instruction exists, searching for alternative data corresponding to the target data in the mock platform based on the URI information;

s503: sending the replacement data to the first system.

As described in the foregoing steps S501 to S503, it is realized that the first system can obtain corresponding substitute data, that is, under the condition that the requirement of the first system on the target data is not strict, for example, only a similar environment needs to be simulated, but the data does not need to be subjected to fine verification, so that there are many data that can meet the requirement of the first system, for example, the data in the second system is the data of the current version, and the data in the first system is the data of the previous version, and if the data of the two versions meet the requirement, the data in the second system is only better, and therefore, when the target data in the second system is not obtained, the substitute data can be sent to the first system. Therefore, target data is replaced, and the efficiency of the first system for acquiring data is further improved.

In one embodiment, before the step S1 of the mock platform receiving a data acquisition request of the first system for a specified item, the mock platform further includes:

s001: acquiring and counting the frequency information of acquiring various data by the first system in a specified time period through an sqoop script;

s002: establishing a data list which needs to be acquired in advance in various data according to the acquired frequency information;

s003: and acquiring and storing corresponding data from a corresponding system based on the data list.

As described in the foregoing steps S001-S003, the Sqoop script is a tool for transferring data in the Hadoop and the relational database to each other, and may import data in a relational database (e.g., MySQL, Oracle, Postgres, etc.) into the HDFS of the Hadoop, and may also import data of the HDFS into the relational database, that is, crawl the mock platform through the Sqoop script to obtain the frequency information of acquiring various data by the corresponding first system. The more times of acquisition, the more frequent the first system acquires the data, so the data can be cached in advance for the first system to acquire. Specifically, the data list may be set according to the number of times, for example, data greater than the set number of times within a specified time.

In an embodiment, after the step S003 of obtaining and saving the corresponding data from the corresponding system based on the data list, the method further includes:

s004: and if the actual use data of the first system to the specified project is received, updating the corresponding data stored in the platform by the actual use data, and forwarding an update notification to the corresponding system.

As described in step S004 above, in order to facilitate the record of the mock platform on the data, the actual usage data may be updated on the corresponding data stored in the platform, and an update notification is forwarded to the corresponding system. If the first system is a test system, the corresponding system is generally a front end, and therefore, the corresponding message is forwarded to the front end, so that the front end and the data in the test system can be unified, and the front end can conveniently perform corresponding tests.

s511: receiving a test request for the specified item from a front end;

s512: splicing the acquired target data and the data acquisition request into a Json file;

s513: and sending the Json file to the front end, and testing the specified project by the front end.

As described in the foregoing steps S511-S513, in the mock platform, the stored data is generally stored in the form of a strip of record, and when a test request of the front end to the specified item is received, the target data and the data acquisition request need to be spliced into a Json file. When the Mock service for the specified item is provided to the front end, whether the item can provide the preset function or not needs to be judged by combining the data acquisition request and the target data. Therefore, after the Mock service platform needs to parse and splice the parameters of the request, the corresponding parameters of the response also need to parse and splice, and then the parameters of the request and the response are spliced into a unified Json file, preferably returned to the front end in an http mode, and the front end performs project testing based on the Json file.

In one embodiment, the step S5 of forwarding the target data to the first system includes:

s521: vectorizing the target data and the data acquisition request to respectively obtain a corresponding data vector and a corresponding request vector;

s522: inputting the data vector and the request vector into a preset decoupling formula to obtain a decoupled data vector;

s523: and sending the data corresponding to the decoupled data vector to the first system.

As described in the above steps S521-S523, the decoupling between the target data and the data acquisition request is achieved. The decoupling mode is not limited, and decoupling between the target data and the data acquisition request can be realized, that is, the coupling between the first system and the second system is further reduced. The vectorization mode is to input the vector machine SVM with the vector machine SVM, and then the corresponding data vector and the corresponding request vector can be obtained. In one particular embodiment, the decoupling formula is

Where α ═ h (| τ |, |), h (| τ |, |) is a magnitude function related to ω and τ,

omega represents a data vector needing decoupling, tau represents a request vector needing decoupling, rho and beta are preset parameters, and theta_(ω，τ)For the angle between ω and τ, | τ | denotes the modulus of the data vector, | ω | denotes the modulus of the request vector, f_d(ω, τ) represents the decoupling function. Therefore, the decoupled data vector is obtained, and the data corresponding to the decoupled data vector is sent to the first system, so that the first system and the second system can be effectively prevented from being over-coupled.

Referring to fig. 2, the present invention also provides a data processing apparatus, including:

a receiving module 10, configured to receive a data acquisition request of a first system for a specified item;

the analysis module 20 is configured to analyze the data acquisition request and acquire URI information of target data required in the data acquisition request;

an obtaining module 30, configured to obtain domain name information of a second system and a URI matching rule with the second system based on the URI information;

a sending module 40, configured to send a request for obtaining the target data to the second system based on the domain name information and the URI matching rule;

a forwarding module 50, configured to forward the target data to the first system when the target data sent by the second system is received.

In one embodiment, the data processing apparatus further comprises:

the target data judgment module is used for judging whether the mock service data contains the target data;

and the target data sending module is used for directly sending the target data in the mock data to the first system if the target data is contained.

In one embodiment, the data processing apparatus further comprises:

the instruction detection module is used for detecting whether the data acquisition request has a confirmation instruction for receiving the data in the mock platform or not if the target data is not returned or the returned data is unavailable by the second system;

the searching module is used for searching for alternative data corresponding to the target data in the mock platform based on the URI information if the confirmation instruction exists;

and the substitute data sending module is used for sending the substitute data to the first system.

In one embodiment, the data processing apparatus further comprises:

the frequency information acquisition module is used for acquiring and counting frequency information of various data acquired by the first system in a specified time period through the sqoop script;

the data list establishing module is used for establishing a data list which needs to be obtained in advance in various data according to the obtained frequency information;

and the storage module is used for acquiring and storing corresponding data from a corresponding system based on the data list.

In one embodiment, the data processing apparatus further comprises:

and the updating module is used for updating the corresponding data stored in the platform by the actual use data and forwarding an updating notice to the corresponding system if the actual use data of the specified project by the first system is received.

In one embodiment, the data processing apparatus further comprises:

the test request receiving module is used for receiving a test request for the specified item from a front end;

the splicing module is used for splicing the acquired target data and the data acquisition request into a Json file;

and the file sending module is used for sending the Json file to the front end, and the front end tests the specified project.

In one embodiment, the forwarding module 50 includes:

the vectorization submodule is used for vectorizing the target data and the data acquisition request to respectively obtain a corresponding data vector and a corresponding request vector;

the decoupling submodule is used for inputting the data vector and the request vector into a preset decoupling formula to obtain a decoupled data vector;

and the data sending submodule is used for sending the data corresponding to the decoupled data vector to the first system.

Referring to fig. 3, a computer device, which may be a server and whose internal structure may be as shown in fig. 3, is also provided in the embodiment of the present application. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer designed processor is used to provide computational and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various URI information and the like. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program may implement the data processing method according to any of the above embodiments when executed by a processor.

Those skilled in the art will appreciate that the architecture shown in fig. 3 is only a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects may be applied.

The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data processing method described in any of the above embodiments can be implemented.

It will be understood by those skilled in the art that all or part of the processes of the methods of the above embodiments may be implemented by hardware associated with instructions of a computer program, which may be stored on a non-volatile computer-readable storage medium, and when executed, may include processes of the above embodiments of the methods. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.

The blockchain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product services layer, and an application services layer.

The block chain underlying platform can comprise processing modules such as user management, basic service, intelligent contract and operation monitoring. The user management module is responsible for identity information management of all blockchain participants, and comprises public and private key generation maintenance (account management), key management, user real identity and blockchain address corresponding relation maintenance (authority management) and the like, and under the authorization condition, the user management module supervises and audits the transaction condition of certain real identities and provides rule configuration (wind control audit) of risk control; the basic service module is deployed on all block chain node equipment and used for verifying the validity of the service request, recording the service request to storage after consensus on the valid request is completed, for a new service request, the basic service firstly performs interface adaptation analysis and authentication processing (interface adaptation), then encrypts service information (consensus management) through a consensus algorithm, transmits the service information to a shared account (network communication) completely and consistently after encryption, and performs recording and storage; the intelligent contract module is responsible for registering and issuing contracts, triggering the contracts and executing the contracts, developers can define contract logics through a certain programming language, issue the contract logics to a block chain (contract registration), call keys or other event triggering and executing according to the logics of contract clauses, complete the contract logics and simultaneously provide the function of upgrading and canceling the contracts; the operation monitoring module is mainly responsible for deployment, configuration modification, contract setting, cloud adaptation in the product release process and visual output of real-time states in product operation, such as: alarm, monitoring network conditions, monitoring node equipment health status, and the like.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A data processing method, comprising:

2. The data processing method of claim 1, wherein mock service data is preset in the mock platform, and before the step of obtaining the domain name information of the second system and the matching rule with the URI of the second system based on the URI information, the method further comprises:

judging whether the mock service data contains the target data or not;

3. The data processing method of claim 1, wherein after the step of sending a request to the second system to obtain the target data based on the domain name information and the URI matching rule, further comprising:

sending the replacement data to the first system.

4. The data processing method of claim 1, wherein the step of the mock platform receiving a data acquisition request of the first system for a specified item is preceded by the step of:

5. The data processing method of claim 4, wherein after the step of obtaining and saving the corresponding data from the corresponding system based on the data list, the method further comprises:

6. The data processing method of claim 1, wherein after the step of sending a request to the second system for obtaining the target data based on the domain name information and the URI matching rule, further comprising:

receiving a test request for the specified item from a front end;

7. The data processing method of claim 1, wherein the step of forwarding the target data to the first system comprises:

8. A data processing apparatus, comprising:

9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.