CN117435628A

CN117435628A - Database transformation method, apparatus, device, medium and program product

Info

Publication number: CN117435628A
Application number: CN202311616057.XA
Authority: CN
Inventors: 魏亚东; 朱宇戈; 刘博�; 张建荣
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-11-29
Filing date: 2023-11-29
Publication date: 2024-01-23

Abstract

The present disclosure provides a database transformation method, which may be applied to the fields of artificial intelligence and financial technology. Comprising the following steps: synchronizing transaction data from a source database to a target database, the transaction data comprising m transaction requests, each transaction request comprising n transaction metrics; respectively counting transaction data in a source database and a target database to obtain a transaction request portrait of the source database and a transaction request portrait of the target database; processing the transaction request portrait of the source database and the transaction request portrait of the target database in real time to obtain a standardized data set, wherein the standardized data set comprises deviation rates of all transaction indexes of the transaction request in the source database and the target database; carrying out abnormal data extraction on the standardized data set by adopting a pre-trained abnormal data extraction model; and performing back-cut on the abnormal data to complete database transformation from the source database to the target database. The present disclosure also provides a database transformation apparatus, device, storage medium, and program product.

Description

Database transformation method, apparatus, device, medium and program product

Technical Field

The present disclosure relates to the field of artificial intelligence and the field of finance, and in particular, to a database transformation method, apparatus, device, medium, and program product.

Background

With the promotion of database localization transformation, in order to keep the tangential flow stable, analysis of transaction request deviation rate is performed on the oracle cluster and the new database cluster, so that positive significance is provided for stable tangential flow production, and the user's look and feel is improved.

At present, the database tangential flow is mainly double writing of the database, and the research and development workload and performance are greatly increased. Meanwhile, the analysis is only dependent on the success rate or failure rate of the service, and the transformation effect of the database cannot be effectively reflected.

Disclosure of Invention

In view of the foregoing, the present disclosure provides database transformation methods, apparatus, devices, media, and program products that improve database transformation efficiency and success rate, for at least partially solving the above technical problems.

According to a first aspect of the present disclosure, there is provided a database transformation method comprising: synchronizing transaction data from a source database to a target database, the transaction data comprising m transaction requests, each transaction request comprising n transaction metrics, m and n being positive integers; respectively counting transaction data in a source database and a target database according to preset time to obtain a source database transaction request portrait and a target database transaction request portrait; processing the transaction request portrait of the source database and the transaction request portrait of the target database in real time to obtain a standardized data set, wherein the standardized data set comprises deviation rates of all transaction indexes of the transaction request in the source database and the target database; under the condition that the deviation rate of the transaction index is larger than a first preset threshold value, adopting a pre-trained abnormal data extraction model to extract abnormal data from the standardized data set; and performing back-cut on the abnormal data to complete database transformation from the source database to the target database.

According to an embodiment of the present disclosure, processing a source database transaction request portrait and a target database transaction request portrait in real time, obtaining a standardized dataset includes: the method comprises the steps of comparing a source database transaction request portrait with a target database transaction request portrait in real time by adopting a distributed computing framework to obtain a comparison result; and (5) carrying out format unification and normalization on the comparison result to obtain a standardized data set.

According to an embodiment of the present disclosure, the transaction index includes a service success rate and a technology success rate, and the real-time comparison between the source database transaction request portrait and the target database transaction request portrait by adopting the distributed computing framework includes: determining a plurality of transaction nodes; comparing the service success rate of the source database with the service success rate of the target database through the transaction node to obtain the deviation rate of the service success rate; comparing the technical success rate of the source database with the technical success rate of the target database through the transaction node to obtain a deviation rate of the technical success rate; wherein the plurality of transaction nodes are nodes for processing the same transaction request.

According to an embodiment of the present disclosure, performing abnormal data extraction on a standardized dataset using a pre-trained abnormal data extraction model includes: and carrying out abnormal data extraction on the standardized data set by adopting a pre-trained isolated forest algorithm model.

According to an embodiment of the present disclosure, performing outlier data extraction on a normalized dataset using a pre-trained isolated forest algorithm model includes: determining the average path length of the deviation rate of the trade index in the isolated forest; determining an anomaly score of the transaction request corresponding to the transaction indicator according to the average path length and the expected average path length; determining that the transaction request is an abnormal transaction request under the condition that the abnormal score is larger than a second preset threshold value; and outputting transaction data corresponding to the abnormal transaction request.

According to an embodiment of the present disclosure, a training method of an isolated forest algorithm model includes: randomly extracting a sub-sample from the normalized dataset; determining the partition attribute of the isolated tree according to any transaction index of the subsamples; determining dividing points of the isolated tree according to the value of any transaction index of the subsamples; dividing the sub-samples into two sub-subsets according to the dividing attribute and the dividing point; repeating the step of dividing the sub-subsets until the sub-samples cannot be divided or reach a preset depth to obtain an isolated tree; and repeating the steps of extracting the subsamples and constructing the isolated trees until the preset number of the isolated trees is reached, thereby obtaining an isolated forest.

According to an embodiment of the present disclosure, performing a loop-back on exception data includes: deleting the abnormal data from the target database; or recovering transaction data corresponding to the abnormal data in the source database.

A second aspect of the present disclosure provides a database transformation apparatus, comprising: the synchronization module is used for synchronizing transaction data from the source database to the target database, wherein the transaction data comprises m transaction requests, each transaction request comprises n transaction indexes, and m and n are positive integers; the statistics module is used for respectively counting transaction data in the source database and the target database according to preset time to obtain a transaction request portrait of the source database and a transaction request portrait of the target database; the processing module is used for processing the transaction request portrait of the source database and the transaction request portrait of the target database in real time to obtain a standardized data set, wherein the standardized data set comprises deviation rates of all transaction indexes of the transaction request in the source database and the target database; the extraction module is used for extracting abnormal data from the standardized data set by adopting a pre-trained abnormal data extraction model under the condition that the deviation rate of the transaction index is larger than a first preset threshold value; and the back-cut module is used for back-cutting the abnormal data to finish database transformation from the source database to the target database.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of the embodiments described above.

A fourth aspect of the present disclosure also provides a computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method of any of the embodiments described above.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the method of any of the embodiments described above.

Compared with the prior art, the database transformation method, the device, the electronic equipment, the storage medium and the program product have at least the following beneficial effects:

(1) According to the database transformation method, the traditional database double-writing method is replaced by the data synchronization method, so that the requirement on double-writing equipment is greatly reduced, and the workload and the equipment cost are further reduced. Meanwhile, abnormal data with overlarge deviation rate can be timely found through real-time monitoring of a plurality of transaction indexes of the synchronous data stream, and relevant data restoration is carried out, so that the integrity of the data is ensured, and the stability of database transformation is further improved.

(2) According to the database transformation method, the distributed framework is adopted to perform extraction on the data stream and calculate the deviation rate of the transaction index in parallel, so that the data processing speed is improved. The comparison results of the transaction indexes are unified and normalized, so that the data can be conveniently subjected to model extraction, and the data processing efficiency is further improved.

(3) The database transformation method specifically adopts the isolated forest algorithm model to extract the abnormal data, and can rapidly extract the abnormal data in the standardized data set aiming at the characteristics of fewer abnormal samples and larger difference of characteristic values in the transformation process, thereby meeting the real-time requirement of the whole database transformation monitoring process.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a database transformation method, apparatus, device, medium and program product according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a database transformation method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart of a method of acquiring a normalized dataset in accordance with an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a method of obtaining a comparison result in accordance with an embodiment of the present disclosure;

FIG. 5 schematically illustrates a method flow diagram for exception data extraction from a standardized dataset in accordance with an embodiment of the present disclosure;

FIG. 6 schematically illustrates a flow chart of a method of exception data extraction for a standardized dataset in accordance with another embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of a method of training an orphan forest algorithm model in accordance with an embodiment of the present disclosure;

FIG. 8 schematically illustrates a method flow diagram for back-cutting exception data, in accordance with an embodiment of the present disclosure;

FIG. 9 schematically illustrates a block diagram of a database transformation apparatus according to an embodiment of the present disclosure; and

fig. 10 schematically illustrates a block diagram of an electronic device adapted to implement a database transformation method according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Embodiments of the present disclosure provide a database transformation method, apparatus, device, medium, and program product, which may be used in the financial field or other fields. It should be noted that the database transformation method, apparatus, device, medium and program product of the present disclosure may be used in the financial field, and may also be used in any field other than the financial field, and the application fields of the database transformation method, apparatus, device, medium and program product of the present disclosure are not limited.

In the technical scheme of the invention, the related user information (including but not limited to user personal information, user image information, user equipment information, such as position information and the like) and data (including but not limited to data for analysis, stored data, displayed data and the like) are information and data authorized by a user or fully authorized by all parties, and the processing of the related data such as collection, storage, use, processing, transmission, provision, disclosure, application and the like are all conducted according to the related laws and regulations and standards of related countries and regions, necessary security measures are adopted, no prejudice to the public welfare is provided, and corresponding operation inlets are provided for the user to select authorization or rejection.

The embodiment of the disclosure provides a database transformation method, which comprises the following steps: synchronizing transaction data from a source database to a target database, the transaction data comprising m transaction requests, each transaction request comprising n transaction metrics, m and n being positive integers; respectively counting transaction data in a source database and a target database according to preset time to obtain a source database transaction request portrait and a target database transaction request portrait; processing the transaction request portrait of the source database and the transaction request portrait of the target database in real time to obtain a standardized data set, wherein the standardized data set comprises deviation rates of all transaction indexes of the transaction request in the source database and the target database; under the condition that the deviation rate of the transaction index is larger than a first preset threshold value, adopting a pre-trained abnormal data extraction model to extract abnormal data from the standardized data set; and performing back-cut on the abnormal data to complete database transformation from the source database to the target database. The database transformation method disclosed by the invention realizes that the analysis of the request deviation rate is completed in the parallel period when the system carries out transaction migration on a plurality of heterogeneous database clusters (such as Oracle migration MySQL, mySQL migration OceanBase), so that the customer perception is minimized, and the production safety is effectively guarded. Meanwhile, the workload of double writing is reduced, and the research and development cost is reduced.

Fig. 1 schematically illustrates an application scenario diagram of a database transformation method, apparatus, device, medium and program product according to an embodiment of the present disclosure.

As shown in fig. 1, an application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device. In particular, server 105 may be a distributed cluster of servers for parallel processing of extraction of data streams and index computation during database transformation. And the server 105 may also be provided with an extraction model of the anomaly data to extract the anomaly data in the data stream.

It should be noted that the database transformation method provided in the embodiments of the present disclosure may be generally performed by the server 105. Accordingly, the database transformation apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 105. The database transformation method provided by the embodiments of the present disclosure may also be performed by a server or a cluster of servers that are different from the server 105 and that are capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the database transformation apparatus provided by the embodiments of the present disclosure may also be provided in a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The database transformation method of the disclosed embodiment will be described in detail below with reference to fig. 2 to 8 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a database transformation method according to an embodiment of the present disclosure.

As shown in fig. 2, the database transformation method of this embodiment includes, for example, operations S210 to S250, and the database transformation method may be executed by a computer program on corresponding computer hardware.

In operation S210, transaction data including m transaction requests, each transaction request including n transaction metrics, m and n being positive integers, are synchronized from a source database to a target database.

In operation S220, transaction data in the source database and the target database are counted according to the preset time, respectively, to obtain a transaction request portrait of the source database and a transaction request portrait of the target database.

In operation S230, the source database transaction request portrayal and the target database transaction request portrayal are processed in real time to obtain a standardized data set including the deviation rates of the transaction indexes of the transaction requests in the source database and the target database.

In operation S240, if the deviation rate of the transaction index is greater than the first preset threshold, the abnormal data extraction is performed on the standardized data set using a pre-trained abnormal data extraction model.

In operation S250, the exception data is cut back to complete the database transformation from the source database to the target database.

For example, there are two databases, one is an existing relational database and the other is a target distributed database. Database transformation is required to migrate transaction data from the source database to the target database. First, transaction data is synchronized from a source database to a target database. These transaction data comprise, for example, 1000 transaction requests, each transaction request comprising 5 transaction metrics. For example, these trade indicators include business success rate, technical success rate, trade time consumption, etc. Next, transaction data in the source database and the target database are counted at preset time intervals (e.g., three dimensions per minute, half hour, hour), respectively. A source database transaction request portrait and a target database transaction request portrait are obtained.

Then, the two images are processed in real time. For example, comprising comparing the values of each transaction index in the source database and the target database and calculating a deviation rate. For example, if the success rate of traffic in the target database is 1% lower than the source database, then the deviation rate is-1%. In this way, a normalized data set is obtained that includes the deviation rate of the transaction metrics for each transaction request in the source and target databases. If the deviation rate of some transaction metrics is greater than a first preset threshold (e.g., 5%), then an anomaly data extraction for this normalized dataset may be performed using a pre-trained anomaly data extraction model. For example, a transaction request is considered anomalous if its technical success rate is 10% lower in the target database than in the source database.

And finally, performing back cut on the extracted abnormal data. Including, for example, switching the exception data back from the target database to the source database, or performing other repair and adjustment operations to ensure that the database transformation from the source database to the target database is performed successfully.

Specifically, information collection can be realized by means of log analysis, a monitoring system, database audit and the like. The result of the information collection is a multi-dimensional dataset, each dimension corresponding to an index, and each data point corresponding to a transaction request. The transaction request representation shown in table 1 was obtained by information acquisition.

TABLE 1 transaction request portrayal

In operation S210, acquiring transaction data of a user involves acquiring personal information of the user.

In embodiments of the present disclosure, the user's consent or authorization may be obtained prior to obtaining the user's information. For example, before operation S210, a request to acquire user information may be issued to the user. In case the user agrees or authorizes that the user information can be acquired, the operation S210 is performed.

Fig. 3 schematically illustrates a flow chart of a method of acquiring a normalized dataset according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, as shown in fig. 3, the source database transaction request portraits and the target database transaction request portraits are processed in real time, for example, by operations S331 to S332, resulting in a standardized data set.

In operation S331, the source database transaction request portrait and the target database transaction request portrait are compared in real time by using the distributed computing framework, and a comparison result is obtained.

In operation S332, the comparison result is subjected to format unification and normalization to obtain a standardized data set.

For example, by analyzing and processing the data stream in real time through streaming processing, rapid response and feedback can be achieved without waiting for the entire arrival of the data. The streaming process may utilize a distributed computing framework to achieve efficient data processing and storage. The result of the streaming process is a real-time updated data set, and each data point contains the comparison result of the transaction request portraits of the source database and the target database, and information such as deviation rate. The average delay of the streaming process is 0.1 seconds, and the hit rate of the cache is 99.9%, so that the consistency and the integrity of the data are ensured.

For example, after transaction request portraits are obtained from the source and target databases, the portraits need to be processed in real-time to obtain a standardized data set. First, a source database transaction request portrait and a target database transaction request portrait are compared in real time using a distributed computing framework, such as Apache Spark. This process can be complicated because each transaction request may include multiple transaction metrics, which require comparison of the values in the source and target databases. For example, the values of the technical success rate of each transaction request in the source database and the target database may be compared. A comparison may be made if the technical success rate of a certain transaction request is 10% lower in the target database than in the source database.

Next, the comparison result needs to be subjected to format unification and normalization so as to facilitate subsequent screening of abnormal data. Normalization refers to converting data into a uniform format and standard, and eliminating differences and conflicts of the data. Normalization refers to converting data into the same range and scale, eliminating the bias and impact of the data. For example, converting all transaction index values to the same format (e.g., converting all values to percentages) and mapping all values to the same scale using a normalization method. For example, all transaction index values may be mapped to a range of 0-1 using a min-max normalization method. The normalized and normalized result is a normalized data set, each data point can be represented by a vector, each component of the vector corresponding to a deviation rate of the index.

Through the data preprocessing process, the real-time processing and storage of the data can be realized, the availability and reliability of the data are improved, and effective input is provided for the subsequent screening of abnormal data, wherein the effective input comprises the deviation rate of each transaction index of each transaction request in a source database and a target database. Through these steps, a normalized data set is obtained, which can be used for further outlier data extraction and cut-back operations.

Fig. 4 schematically illustrates a flow chart of a method of obtaining a comparison result in accordance with an embodiment of the present disclosure.

According to embodiments of the present disclosure, the transaction metrics include, for example, business success rate and technical success rate. As shown in fig. 4, for example, the source database transaction request portrait and the target database transaction request portrait are compared in real time by operations S4311 to S4313, and a comparison result is obtained.

In operation S4311, a plurality of transaction nodes is determined. Wherein the plurality of transaction nodes are nodes for processing the same transaction request.

In operation S4312, the business success rate of the source database is compared with the business success rate of the target database through the transaction node, and a deviation rate of the business success rate is obtained.

In operation S4313, the technical success rate of the source database is compared with the technical success rate of the target database through the transaction node, and a deviation rate of the technical success rate is obtained.

For example, a database transformation case of a financial services company is processed. This company uses a traditional relational database, but business development requires a more efficient distributed database to support. During the transaction, two key indicators need to be paid attention to: service success rate and technical success rate. First, the source database transaction request portraits and the target database transaction request portraits are compared in real-time using a distributed computing framework, such as Apache Spark. In this process, a plurality of transaction nodes are determined, which are nodes that process the same transaction request.

For example, assume that a transaction request for an online payment is being processed. This transaction request may pass through a number of nodes including user interfaces, payment processing, and settlement, etc. Each node generates a transaction indicator such as a business success rate and a technology success rate. And comparing the service success rate of the source database with the service success rate of the target database through the transaction node to obtain the deviation rate of the service success rate. For example, if the traffic success rate of the source database is 95% and the traffic success rate of the target database is 90%, the deviation rate of the traffic success rate is 5%. Similarly, the technical success rate of the source database and the technical success rate of the target database can be compared through the transaction node, so that the deviation rate of the technical success rate is obtained. For example, if the technical success rate of the source database is 90% and the technical success rate of the target database is 85%, the deviation rate of the technical success rate is 5%.

Through the steps, a detailed comparison result can be obtained, wherein the detailed comparison result comprises the deviation rate of the service success rate and the technical success rate of each transaction node. The multi-dimensional real-time monitoring of transaction data synchronization is realized by comparing the success rate of the parallel processing business with the success rate of the technology by a plurality of nodes. Such information may help to better understand problems and challenges that may exist in the migration process from the source database to the target database, and to take appropriate action to address such problems.

FIG. 5 schematically illustrates a flow chart of a method of exception data extraction for a standardized dataset in accordance with an embodiment of the present disclosure.

According to an embodiment of the present disclosure, as shown in fig. 5, abnormal data extraction is performed on the standardized data set, for example, through operation S541.

In operation S541, abnormal data extraction is performed on the standardized dataset using a pre-trained isolated forest algorithm model.

For example, an isolated forest algorithm model is used for anomaly data extraction. An isolated forest is an unsupervised machine learning algorithm that can be used for anomaly detection tasks. First, a standardized dataset is input into a pre-trained isolated forest algorithm model. This model evaluates the degree of anomaly for each transaction request based on the deviation rate for each transaction indicator. For example, if the transaction success rate of a certain transaction request is 10% lower in the target database than in the source database, while the transaction success rate of other transaction requests are within a reasonable range, then the transaction request may be considered anomalous. The isolated forest algorithm model extracts abnormal data according to the degree of abnormality of each transaction request. These data may be further analyzed and processed to determine whether a cut-back operation or other repair and adjustment operations are required. The average execution time of the isolated forest algorithm is 0.01 second, the accuracy of anomaly detection is 98.7%, the success rate of back cut is 100%, the customer perception is 0.01%, and the production safety is guaranteed.

It will be appreciated that this is merely a specific example and that actual operation may need to be adapted and optimized to specific service requirements and data conditions. In addition, the isolated forest algorithm is only one of the feasible abnormal data extraction methods, and other algorithms or models can be selected according to specific situations to extract abnormal data.

It should be noted that, before the abnormal data extraction is performed on the standardized data set by using the pre-trained isolated forest algorithm model, the relationship between the deviation rate of the transaction index and the first preset threshold value is also required to be determined. The first preset threshold is set to (-5%, 5%), for example, and then an isolated forest algorithm is used to quickly derive anomalies and make a cut back, otherwise the transaction is considered normal.

FIG. 6 schematically illustrates a flow chart of a method of exception data extraction for a standardized dataset in accordance with another embodiment of the present disclosure.

According to an embodiment of the present disclosure, as shown in fig. 6, abnormal data extraction is performed on the standardized data set, for example, through operations S6411 to S6414.

In operation S6411, an average path length of the deviation rate of the trade index in the isolated forest is determined.

In operation S6412, an anomaly score of the transaction request corresponding to the transaction index is determined from the average path length and the expected average path length.

In operation S6413, in the case where the anomaly score is greater than the second preset threshold value, it is determined that the transaction request is an anomalous transaction request.

In operation S6414, transaction data corresponding to the abnormal transaction request is output.

For example, first, it is necessary to determine the average path length of the deviation rate of the trade index in the isolated forest. This step is to calculate the degree of abnormality of each transaction index. For example, the degree of anomaly for each trading index can be determined by calculating the average path length for all trading index deviation rates. If the deviation rate of a certain trade index is far higher than the average level, then the trade request corresponding to the trade index may be abnormal. Next, an anomaly score for the transaction request corresponding to the transaction indicator needs to be determined based on the average path length and the expected average path length. This step is to evaluate each transaction request for anomalies. For example, if the deviation rate of a transaction indicator is much higher than the expected average path length, the anomaly score for the transaction request will be higher. This score may represent the degree of abnormality of this transaction request.

Wherein, the anomaly score refers to an index for measuring the anomaly degree of the data point, and the closer to 1 is more abnormal, and the closer to 0 is more normal. The process of calculating the anomaly score is as follows: for each data point, its path length in each orphan tree, i.e. the number of divisions from the root node to the leaf nodes, is calculated. For each data point, its average path length in the isolated forest, i.e. the average of the path lengths in all the isolated trees, is calculated. Based on its average path length and the expected average path length, its anomaly score is calculated as follows:

s(x，n)＝2-c(n)E(h(x)) (1)

c(n)＝2H(n-1)-(2(n-1)/n) (2)

Where x is the data point, n is the size of the subsamples, h (x) is the average path length of the data point, E (h (x)) is the expected average path length, and c (n) is a constant used to normalize the anomaly score.

Then, in the event that the anomaly score is greater than a second preset threshold, this transaction request may be determined to be an anomalous transaction request. This step is to screen out abnormal transaction requests. For example, a transaction request may be considered anomalous if its anomaly score exceeds a second predetermined threshold (e.g., 0.8). These abnormal transaction requests require further processing and repair.

Finally, transaction data corresponding to the abnormal transaction request needs to be output. These data can be used for further analysis and processing. For example, the specifics of these abnormal transaction requests may be reviewed to see why they may be considered abnormal. These data may also be provided to other systems or teams for further processing and decision making.

Through the steps, the isolated forest algorithm model can be used for extracting abnormal data from the standardized data set and outputting transaction data corresponding to abnormal transaction requests. These data can be used for further business analysis and processing to help better understand the problems and challenges presented in database transformation.

Fig. 7 schematically illustrates a flow chart of a method of training an orphan forest algorithm model according to an embodiment of the disclosure.

According to an embodiment of the present disclosure, an orphan forest algorithm model is trained, for example, by operations S7411 to S7416, as shown in fig. 7.

In operation S7411, a sub-sample is randomly extracted from the normalized dataset.

In operation S7412, the partition attribute of the orphan tree is determined based on any transaction index of the subsamples.

In operation S7413, the dividing points of the orphan tree are determined based on the value of any transaction index of the subsamples.

In operation S7414, the sub-samples are divided into two sub-subsets according to the division attribute and the division point.

In operation S7415, the step of dividing the sub-sets is repeated until the sub-samples cannot be divided or reach a preset depth, resulting in an isolated tree.

In operation S7416, the steps of extracting the subsamples and constructing the isolated trees are repeated until a predetermined number of isolated trees are reached, resulting in an isolated forest.

For example, an isolated forest is a collection of multiple isolated trees, each of which is a binary tree, used to randomly partition a data space. The process of constructing an isolated forest is as follows: first, a sub-sample needs to be randomly extracted from the normalized dataset. This subsampled is the dataset used to construct the orphan tree. For example, 100 transaction requests may be randomly selected from the standardized dataset as sub-samples. These transaction requests are data points used to build an orphan tree. Next, the partition attribute of the orphan tree needs to be determined according to any transaction index of the subsamples. This step is to select attributes for partitioning the data. One dimension is randomly selected from all dimensions of the data as a partitioning attribute of the orphan tree, for example, a business success rate may be selected as a partitioning attribute because this index is important in the transaction process.

Then, the dividing points of the isolated tree need to be determined according to the value of any transaction index of the subsamples. This step is to select points that divide the data points into two sub-subsets. A value is randomly selected from between the maximum value and the minimum value of the dimension as a dividing point of the isolated tree, for example, 50% of the success rate of the service may be selected as the dividing point. Next, the sub-samples need to be divided into two sub-subsets according to the division attribute and the division point, as the left and right sub-trees of the isolated tree. This step is to divide the data set into two parts. For example, transaction requests with a service success rate higher than 50% are divided into one sub-set a, and transaction requests with a service success rate lower than 50% are divided into another sub-set B.

Then, the sub-subset dividing step is repeated until the sub-samples cannot be divided or reach a preset depth, and an isolated tree is obtained. This step is to build an orphan tree. For example, the partitioning of the sub-subsets a and B may continue until there are only 1 data point in each sub-subset or a preset maximum depth is reached. Thus we have obtained an isolated tree.

And finally, repeating the steps of extracting the subsamples and constructing the isolated trees until the preset number of the isolated trees is reached, and obtaining the isolated forest. This step is to build multiple isolated trees and form an isolated forest. For example, the above steps may be repeated 1000 times, each time using a different subsamples to construct an orphan tree, until 1000 orphan trees are obtained. Thus, an isolated forest of 1000 isolated trees was obtained.

Through the steps, the isolated forest algorithm model which can be used for extracting the abnormal data can be trained. This model may be used to perform anomalous data extraction from the standardized data set and output transaction data corresponding to anomalous transaction requests.

FIG. 8 schematically illustrates a flow chart of a method of cutting back abnormal data in accordance with an embodiment of the present disclosure.

According to an embodiment of the present disclosure, as shown in fig. 8, abnormal data is cut back, for example, through operations S851 to S852.

In operation S851, the abnormal data is deleted from the target database. Or alternatively

In operation S852, transaction data corresponding to the abnormal data in the source database is restored.

For example, the exception data may be deleted from the target database. This step is to clear the exception data to avoid further impact on the target database. Transaction request data identified as anomalous by the orphan forest algorithm model may be deleted from the target database. This ensures that the target database does not contain abnormal data.

In addition, transaction data corresponding to the abnormal data in the restoration source database may be selected. This step is to restore the exception data into the source database to preserve the integrity of the source database. For example, data sources such as backups or audit logs may be used to find transaction data corresponding to the anomalous data and restore it to the source database. This ensures data consistency and integrity of the source database.

It will be appreciated that the specific operation of cutting back the exception data depends on the business requirements and the actual situation. In some cases, both methods of deletion and restoration may be used simultaneously to handle anomalous data. Meanwhile, in order to ensure data security and accuracy, necessary backup and verification work may be performed before the back-cut operation.

Through the steps, abnormal data can be cut back so as to maintain the integrity and consistency of the database. These operations can effectively handle anomalous data and protect the data assets of the enterprise.

Based on the database transformation method, the disclosure also provides a database transformation device. The database transformation apparatus will be described in detail with reference to fig. 9.

Fig. 9 schematically shows a block diagram of a database transformation apparatus according to an embodiment of the present disclosure.

As shown in fig. 9, the database transformation apparatus 900 of this embodiment includes, for example: a synchronization module 910, a statistics module 920, a processing module 930, an extraction module 940, and a cut-back module 950.

The synchronization module 910 is configured to synchronize transaction data from a source database to a target database, the transaction data including m transaction requests, each transaction request including n transaction metrics, where m and n are positive integers. In an embodiment, the synchronization module 910 may be configured to perform the operation S210 described above, which is not described herein.

The statistics module 920 is configured to respectively count transaction data in the source database and the target database according to a preset time, so as to obtain a transaction request portrait of the source database and a transaction request portrait of the target database. In an embodiment, the statistics module 920 may be used to perform the operation S220 described above, which is not described herein.

The processing module 930 is configured to process the source database transaction request portrait and the target database transaction request portrait in real time to obtain a standardized data set, where the standardized data set includes deviation rates of transaction indexes of the transaction request in the source database and the target database. In an embodiment, the processing module 930 may be configured to perform the operation S230 described above, which is not described herein.

The extraction module 940 is configured to perform abnormal data extraction on the standardized data set using a pre-trained abnormal data extraction model if the deviation rate of the transaction index is greater than a first preset threshold. In an embodiment, the extracting module 940 may be configured to perform the operation S240 described above, which is not described herein.

The loop-back module 950 is configured to loop back the abnormal data to complete database transformation from the source database to the target database. In an embodiment, the back-cut module 950 may be used to perform the operation S250 described above, which is not described herein.

Any of the synchronization module 910, the statistics module 920, the processing module 930, the extraction module 940, and the cut-back module 950 may be combined in one module to be implemented, or any of the modules may be split into multiple modules, according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. According to embodiments of the present disclosure, at least one of the synchronization module 910, the statistics module 920, the processing module 930, the extraction module 940, and the cut-back module 950 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system-on-chip, a system-on-substrate, a system-on-package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware. Alternatively, at least one of the synchronization module 910, the statistics module 920, the processing module 930, the extraction module 940, and the cut-back module 950 may be at least partially implemented as computer program modules, which when executed, may perform the corresponding functions.

As shown in fig. 10, an electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1002 or a program loaded from a storage section 1008 into a Random Access Memory (RAM) 1003. The processor 1001 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 1001 may also include on-board memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the present disclosure.

In the RAM 1003, various programs and data necessary for the operation of the electronic apparatus 1000 are stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other by a bus 1004. The processor 1001 performs various operations of the method flow according to the embodiment of the present disclosure by executing programs in the ROM 1002 and/or the RAM 1003. Note that the program may be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 may also perform various operations of the method flow according to the embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the disclosure, the electronic device 1000 may also include an input/output (I/O) interface 1005, the input/output (I/O) interface 1005 also being connected to the bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I/O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output portion 1007 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage portion 1008 including a hard disk or the like; and a communication section 1009 including a network interface card such as a LAN card, a modem, or the like. The communication section 1009 performs communication processing via a network such as the internet. The drive 1010 is also connected to the I/O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in the drive 1010, so that a computer program read out therefrom is installed as needed in the storage section 1008.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium described above carries one or more programs, which when executed, implement a database transformation method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 1002 and/or RAM1003 and/or one or more memories other than ROM 1002 and RAM1003 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code means for causing a computer system to carry out the database transformation method provided by the embodiments of the present disclosure when the computer program product is run on the computer system.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1001. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of signals on a network medium, distributed, and downloaded and installed via the communication section 1009, and/or installed from the removable medium 1011. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 1009, and/or installed from the removable medium 1011. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1001. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A method of database transformation comprising:

synchronizing transaction data from a source database to a target database, the transaction data comprising m transaction requests, each transaction request comprising n transaction metrics, m and n being positive integers;

Respectively counting the transaction data in the source database and the target database according to preset time to obtain a source database transaction request portrait and a target database transaction request portrait;

processing the transaction request portrait of the source database and the transaction request portrait of the target database in real time to obtain a standardized data set, wherein the standardized data set comprises the deviation rate of each transaction index of the transaction request in the source database and the target database;

under the condition that the deviation rate of the transaction index is larger than a first preset threshold value, adopting a pre-trained abnormal data extraction model to extract abnormal data from the standardized data set; and

and performing back-cut on the abnormal data to complete database transformation from the source database to the target database.

2. The method of claim 1, wherein processing the source database transaction request portrait and the target database transaction request portrait in real time to obtain a standardized dataset comprises:

the distributed computing framework is adopted to conduct real-time comparison on the source database transaction request portrait and the target database transaction request portrait, and a comparison result is obtained;

And carrying out format unification and normalization on the comparison result to obtain the standardized data set.

3. The method of claim 2, wherein the transaction indicator comprises a business success rate and a technology success rate, and wherein the employing a distributed computing framework to compare the source database transaction request representation and the target database transaction request representation in real time, the obtaining a comparison result comprises:

determining a plurality of transaction nodes;

comparing the service success rate of the source database with the service success rate of the target database through the transaction node to obtain a deviation rate of the service success rate; and

comparing the technical success rate of the source database with the technical success rate of the target database through the transaction node to obtain a deviation rate of the technical success rate;

wherein the plurality of transaction nodes are nodes that process the same transaction request.

4. The method of claim 1, wherein the performing anomaly data extraction on the standardized dataset using a pre-trained anomaly data extraction model comprises:

and carrying out abnormal data extraction on the standardized data set by adopting a pre-trained isolated forest algorithm model.

5. The method of claim 4, wherein the performing outlier data extraction on the normalized dataset using a pre-trained orphan forest algorithm model comprises:

determining an average path length of the deviation rate of the trade index in the isolated forest;

determining an anomaly score of the transaction request corresponding to the transaction indicator according to the average path length and the expected average path length;

determining that the transaction request is an abnormal transaction request under the condition that the abnormal score is larger than a second preset threshold value; and

outputting the transaction data corresponding to the abnormal transaction request.

6. The method of claim 4, wherein the training method of the orphan forest algorithm model comprises:

randomly extracting a sub-sample from said normalized dataset;

determining the partition attribute of the isolated tree according to any transaction index of the subsamples;

determining dividing points of the isolated tree according to the value of any transaction index of the subsamples;

dividing the sub-samples into two sub-subsets according to the dividing attribute and the dividing point;

repeating the step of dividing the sub-subset until the sub-sample cannot be divided or reaches a preset depth to obtain the isolated tree; and

Repeating the steps of extracting subsamples and constructing the isolated trees until a preset number of isolated trees are reached, and obtaining the isolated forest.

7. The method of claim 1, wherein the back-cutting the exception data comprises:

deleting the abnormal data from the target database; or alternatively

And recovering the transaction data corresponding to the abnormal data in the source database.

8. A database transformation apparatus, comprising:

the system comprises a synchronization module, a target database and a source database, wherein the synchronization module is used for synchronizing transaction data from the source database to the target database, the transaction data comprises m transaction requests, each transaction request comprises n transaction indexes, and m and n are positive integers;

the statistics module is used for respectively counting the transaction data in the source database and the target database according to preset time to obtain a transaction request portrait of the source database and a transaction request portrait of the target database;

the processing module is used for processing the transaction request portrait of the source database and the transaction request portrait of the target database in real time to obtain a standardized data set, wherein the standardized data set comprises the deviation rate of each transaction index of the transaction request in the source database and the target database;

The extraction module is used for extracting abnormal data from the standardized data set by adopting a pre-trained abnormal data extraction model under the condition that the deviation rate of the transaction index is larger than a first preset threshold value; and

and the back-cut module is used for back-cutting the abnormal data to finish database transformation from the source database to the target database.

9. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-7.

10. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1-7.

11. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.