CN117522349A - Automatic processing method, equipment and medium for multi-source data service - Google Patents
Automatic processing method, equipment and medium for multi-source data service Download PDFInfo
- Publication number
- CN117522349A CN117522349A CN202410008109.3A CN202410008109A CN117522349A CN 117522349 A CN117522349 A CN 117522349A CN 202410008109 A CN202410008109 A CN 202410008109A CN 117522349 A CN117522349 A CN 117522349A
- Authority
- CN
- China
- Prior art keywords
- processing
- data
- target
- abnormal
- flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 241
- 230000002159 abnormal effect Effects 0.000 claims abstract description 93
- 238000000034 method Methods 0.000 claims abstract description 57
- 238000012423 maintenance Methods 0.000 claims abstract description 29
- 239000011159 matrix material Substances 0.000 claims description 40
- 238000013507 mapping Methods 0.000 claims description 36
- 238000012986 modification Methods 0.000 claims description 8
- 230000004048 modification Effects 0.000 claims description 8
- 230000005856 abnormality Effects 0.000 claims description 5
- 238000012937 correction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 12
- 238000012216 screening Methods 0.000 description 10
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012367 process mapping Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application relates to the field of electric digital data processing, and particularly discloses an automatic processing method, equipment and medium for multi-source data service, wherein the method comprises the following steps: determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively; receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow; selecting target data from a target data source to process based on a preset time interval; if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs and call records of business call interfaces corresponding to the business processing sub-flows respectively; and sending the processing log and the call record to an operation and maintenance personnel. The failure record is compensated through the timing task, and the result of the normally performed record is processed, so that the labor cost of operation and maintenance can be reduced.
Description
Technical Field
The present invention relates to the field of electronic digital data processing, and in particular, to an automated processing method, apparatus, and medium for multi-source data service.
Background
Multi-source data services refer to services involving interfaces of multiple data sources that require frequent processing of data corresponding to different data sources. Taking a risk screening service as an example, after a client submits information of a insured person in the prior art, the risk screening service is required to be carried out, and the corresponding flow of the risk screening service is manually and asynchronously processed, so that the service is interrupted due to the fact that calling failure such as network abnormality exists in a multiparty data source interface, and a certain labor cost exists because the problem is required to be examined and the risk screening service is required to be carried out again.
In order to improve service efficiency, reduce labor cost and save service processing time, and solve the problem of service interruption caused by failure of data source call, an automatic processing method for multi-source data service is needed.
Disclosure of Invention
In order to solve the above problems, the present application proposes a method, an apparatus, and a medium, wherein the method includes:
determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively; receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow; selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface; if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface; and sending the processing log and the call record to an operation and maintenance personnel.
In one example, the receiving the processing exception information from the business processing sub-flow specifically includes: determining current input data, current output data, historical input data and historical output data of the business processing sub-process; determining a mapping similarity of a historical processing mapping and a current processing mapping based on the historical input data, the historical output data, the current input data and the current output data; if the mapping similarity is lower than a preset threshold, a mapping modification log corresponding to the business processing sub-flow is obtained; and based on the mapping modification log and the mapping similarity, performing abnormality judgment on the output data corresponding to the business processing sub-flow.
In one example, the determining the mapping similarity between the historical processing mapping and the current processing mapping specifically includes: converting the historical input data, the historical output data, the current input data and the current output data into a first matrix, a second matrix, a third matrix and a fourth matrix respectively based on the number of data sources; the matrix dimension of the first matrix and the matrix dimension of the third matrix are equal to the number of the data sources; taking the third matrix as input data, and mapping to obtain a fifth matrix through historical processing; and determining the matrix similarity between the fourth matrix and the fifth matrix, and taking the matrix similarity as the mapping similarity of the historical processing mapping and the current processing mapping.
In one example, the sending the processing log and the call record to the operation and maintenance personnel specifically includes: acquiring a history processing result of the operation and maintenance personnel on a history abnormal business processing sub-process; determining the flow similarity of the current abnormal business processing sub-flow and the historical abnormal business processing sub-flow based on the processing log and the calling record; determining the target historical abnormal business processing sub-flow with highest flow similarity corresponding to the current abnormal business processing sub-flow; and sending the history processing result, the processing log and the calling record of the target history abnormal business processing sub-process to operation and maintenance personnel.
In one example, the method further comprises, prior to reselecting the target data among the target data sources based on a preset time interval: acquiring the processing abnormal times, preset priority levels and the number of current idle threads corresponding to the abnormal service processing sub-flow; and determining an abnormal processing time interval of the abnormal business processing sub-process based on the abnormal processing times, the preset priority level and the current idle thread number.
In one example, the determining the abnormal processing time interval of the abnormal service processing sub-process based on the abnormal processing times, the preset priority level and the current idle thread number specifically includes: the exception handling time interval is determined by the following formula:
wherein,for exception handling time interval, +.>To deal with the number of exceptions, +.>For correction factor +.>For the number of data sources corresponding to the abnormal business processing sub-flow,/-for the abnormal business processing sub-flow>For the corresponding input data quantity in the ith data source of the abnormal business processing sub-flow,/the sub-flow is processed>For preset priority level, ++>Is the current number of idle threads.
In one example, before the sending the processing log and the call record to the operation and maintenance personnel, the method further includes: determining a service data set corresponding to the abnormal service processing sub-flow; and determining user private data in the service data set, and encrypting the user private data.
In one example, after the processing the target data through the target service call interface, the method specifically includes: determining that the processing result of the target service on the target data is normal processing; and recording the normal processing result of the abnormal business processing sub-flow.
The application also provides an automated processing device for multi-source data service, comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform: determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively; receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow; selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface; if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface; and sending the processing log and the call record to an operation and maintenance personnel.
The present application also provides a non-volatile computer storage medium storing computer-executable instructions configured to: determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively; receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow; selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface; if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface; and sending the processing log and the call record to an operation and maintenance personnel.
The method provided by the application has the following beneficial effects: and carrying out automatic compensation processing on the failure record through a timing task, and carrying out result processing on the normally carried out record. Meanwhile, an operation and maintenance person can check the log record to conduct problem investigation on records which are unsuccessful in automatic compensation processing, check calling conditions of all interfaces, conduct software automatic processing on risk screening business, and therefore labor cost of operation and maintenance can be greatly reduced.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
fig. 1 is a schematic flow chart of an automated processing method for a multi-source data service in an embodiment of the present application;
fig. 2 is a schematic structural diagram of an automated processing apparatus for multi-source data service according to an embodiment of the present application.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.
Fig. 1 is a flow chart of an automated processing method for multi-source data services according to one or more embodiments of the present disclosure. The method can be applied to different business fields, such as the internet financial business field, the electric business field, the instant messaging business field, the game business field, the public business field and the like. The process may be performed by a computing device in the corresponding domain (e.g., a wind control server or intelligent mobile terminal corresponding to the payment service, etc.), and certain input parameters or intermediate results in the process allow for manual intervention adjustments to help improve accuracy.
The implementation of the analysis method according to the embodiment of the present application may be a terminal device or a server, which is not particularly limited in this application. For ease of understanding and description, the following embodiments are described in detail with reference to a server.
It should be noted that the server may be a single device, or may be a system composed of a plurality of devices, that is, a distributed server, which is not specifically limited in this application.
As shown in fig. 1, an embodiment of the present application provides an automated processing method for a multi-source data service, including:
s101: and determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively.
Firstly, determining a plurality of data sources corresponding to a target multi-source data service and service call interfaces corresponding to a plurality of service processing sub-flows respectively, taking the target multi-source data service as a risk screening service for example, wherein when the risk screening is carried out, a plurality of data are needed for screening a protected person, such as personal information, income information and the like, the service processing sub-flows refer to all sub-flows when the risk screening is carried out, call sequences exist among the sub-flows, namely, after the current sub-flow is processed, the next sub-flow can be processed, and data correlation can exist among all sub-flows, namely, the output result of the current sub-flow is input data of the next sub-flow.
S102: and receiving processing abnormality information from the business processing sub-flow, and determining a target data source and a target business calling interface corresponding to the abnormal business processing sub-flow.
Because of the multi-party data source interface, there may be a call failure such as network exception, which leads to service interruption, and at this time, the service processing sub-flow may send processing exception information to the server, and at this time, the server may determine, based on the processing exception information, the target data source and the target service call interface corresponding to the abnormal service processing sub-flow. For example, the risk screening service is abnormal in the step of confirming the character information, and at this time, the target data source should be confirmed to be the data source and the calling interface corresponding to the identity information.
In one embodiment, before receiving the business anomaly information, the server may determine whether the business data is normal, where it is required to determine current input data, current output data, historical input data, and historical output data of the business process sub-flow, and then determine a mapping similarity of the historical process mapping and the current process mapping based on the historical input data, the historical output data, the current input data, and the current output data. If the mapping similarity is lower than a preset threshold, a mapping modification log corresponding to the business processing sub-flow is obtained, and based on the mapping modification log and the mapping similarity, the output data corresponding to the business processing sub-flow is subjected to abnormality judgment.
Further, when the mapping similarity judgment is performed, the historical input data, the historical output data, the current input data and the current output data need to be respectively converted into a first matrix, a second matrix, a third matrix and a fourth matrix based on the number of data sources; the first matrix and the third matrix have the same matrix dimension as the number of data sources; and then taking the third matrix as input data, mapping through historical processing to obtain a fifth matrix, determining the similarity of the fourth matrix and the fifth matrix, and taking the similarity of the matrix as the mapping similarity of the historical processing mapping and the current processing mapping.
S103: and re-selecting target data in the target data source based on a preset time interval, and processing the target data through the target service calling interface.
When the service is abnormal and is interrupted, the compensation mechanism can resend the data, at the moment, the target data is selected in the target data source again based on a preset time interval, and the target data is processed through the target service calling interface.
In one embodiment, if the preset time interval is set to be shorter, when compensating the abnormal service processing sub-flow, the problems of excessive occupied threads, data blocking and the like caused by more compensation times are encountered, so that different time intervals can be confirmed according to the current situation of the abnormal service processing sub-flow. At this time, the number of processing exceptions, preset priority and the number of current idle threads corresponding to the abnormal business processing sub-flow are required to be obtained; and then determining an abnormal processing time interval of the abnormal business processing sub-flow based on the abnormal processing times, the preset priority level and the current idle thread number.
Further, in determining the exception handling time interval, the validation may be performed by the following formula:
wherein,for exception handling time interval, +.>To deal with the number of exceptions, +.>For correction factor +.>For the number of data sources corresponding to the abnormal business processing sub-flow,/-for the abnormal business processing sub-flow>For the corresponding input data quantity in the ith data source of the abnormal business processing sub-flow,/the sub-flow is processed>For preset priority level, ++>Is the current number of idle threads.
S104: and if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface.
And after multiple failures, the data is subjected to service suspension, and at the moment, processing logs and call records of service call interfaces corresponding to the multiple service processing sub-flows are required to be acquired.
S105: and sending the processing log and the call record to an operation and maintenance personnel.
After the processing log and the call record are sent to the operation and maintenance personnel, the operation and maintenance personnel select manual retransmission after the problem is checked and processed, and the service flow of the data is restarted.
In one embodiment, when the processing log and the call record are sent to the operation and maintenance personnel, the operation and maintenance personnel can send the history cases and the history processing results similar to the current abnormal business processing sub-process together. At the moment, the historical processing result of the operation and maintenance personnel on the historical abnormal business processing sub-flow is required to be obtained, the flow similarity between the current abnormal business processing sub-flow and the historical abnormal business processing sub-flow is determined based on the processing log and the calling record, then the target historical abnormal business processing sub-flow with the highest flow similarity corresponding to the current abnormal business processing sub-flow is determined, and then the historical processing result, the processing log and the calling record of the target historical abnormal business processing sub-flow are sent to the operation and maintenance personnel.
The above-mentioned history abnormal business processing sub-flow and history processing result can be stored in the storage device of the computer equipment in advance, when the process similarity needs to be determined, the computer equipment can select the history abnormal business processing sub-flow and history processing result from the storage device. Of course, the computer device may also obtain the historical abnormal business processing sub-flow and the historical processing result from other external devices. For example, the historical abnormal business processing sub-process and the historical processing result are stored in the cloud, when the process similarity needs to be determined, the computer device can obtain the historical abnormal business processing sub-process and the historical processing result from the cloud, and the obtaining mode of the historical abnormal business processing sub-process and the historical processing result is not limited in this embodiment.
In one embodiment, the business data is also encrypted before the call record is sent to the operation and maintenance personnel to prevent user privacy disclosure. At this time, a service data set corresponding to the abnormal service processing sub-flow needs to be determined, then user private data is determined in the service data set, and the user private data is encrypted.
In one embodiment, after the target data is processed through the target service call interface, if it is determined that the processing result of the target service on the target data is normal processing, the normal processing result of the abnormal service processing sub-flow may be recorded.
The traditional business process is manually processed by manpower, and has the conditions of low efficiency, high labor cost and long processing feedback period. The invention carries out automatic upgrading on the software, processes the business data by the software, records the processing condition of each flow link, captures the abnormal condition, carries out compensation processing on the failure record at regular time, and simultaneously can accurately lock the problem source by only checking the log by operation and maintenance personnel. And the software can encrypt the service data, so that the user privacy can be fully protected.
As shown in fig. 2, the embodiment of the present application further provides an automated processing apparatus for a multi-source data service, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to:
determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively; receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow; selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface; if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface; and sending the processing log and the call record to an operation and maintenance personnel.
The embodiments also provide a non-volatile computer storage medium storing computer executable instructions configured to:
determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively; receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow; selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface; if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface; and sending the processing log and the call record to an operation and maintenance personnel.
All embodiments in the application are described in a progressive manner, and identical and similar parts of all embodiments are mutually referred, so that each embodiment mainly describes differences from other embodiments. In particular, for the apparatus and medium embodiments, the description is relatively simple, as it is substantially similar to the method embodiments, with reference to the section of the method embodiments being relevant.
The devices and media provided in the embodiments of the present application are in one-to-one correspondence with the methods, so that the devices and media also have similar beneficial technical effects as the corresponding methods, and since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media are not described in detail herein.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (10)
1. An automated processing method for multi-source data service, comprising:
determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively;
receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow;
selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface;
if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface;
and sending the processing log and the call record to an operation and maintenance personnel.
2. The method of claim 1, wherein the receiving the processing exception information from the business processing sub-process specifically comprises:
determining current input data, current output data, historical input data and historical output data of the business processing sub-process;
determining a mapping similarity of a historical processing mapping and a current processing mapping based on the historical input data, the historical output data, the current input data and the current output data;
if the mapping similarity is lower than a preset threshold, a mapping modification log corresponding to the business processing sub-flow is obtained;
and based on the mapping modification log and the mapping similarity, performing abnormality judgment on the output data corresponding to the business processing sub-flow.
3. The method according to claim 2, wherein determining the mapping similarity of the historical processing map and the current processing map specifically comprises:
converting the historical input data, the historical output data, the current input data and the current output data into a first matrix, a second matrix, a third matrix and a fourth matrix respectively based on the number of data sources; the matrix dimension of the first matrix and the matrix dimension of the third matrix are equal to the number of the data sources;
taking the third matrix as input data, and mapping to obtain a fifth matrix through historical processing;
and determining the matrix similarity between the fourth matrix and the fifth matrix, and taking the matrix similarity as the mapping similarity of the historical processing mapping and the current processing mapping.
4. The method according to claim 1, wherein said sending the processing log and the call record to an operation and maintenance person specifically comprises:
acquiring a history processing result of the operation and maintenance personnel on a history abnormal business processing sub-process;
determining the flow similarity of the current abnormal business processing sub-flow and the historical abnormal business processing sub-flow based on the processing log and the calling record;
determining a target historical abnormal business processing sub-flow with highest flow similarity corresponding to the current abnormal business processing sub-flow;
and sending the history processing result, the processing log and the calling record of the target history abnormal business processing sub-process to operation and maintenance personnel.
5. The method of claim 1, wherein the method further comprises, prior to reselecting the target data in the target data source based on the preset time interval:
acquiring the processing abnormal times, preset priority levels and the number of current idle threads corresponding to the abnormal business processing sub-processes;
and determining an abnormal processing time interval of the abnormal business processing sub-process based on the abnormal processing times, the preset priority level and the current idle thread number.
6. The method according to claim 5, wherein determining the exception handling time interval of the exception handling sub-process based on the number of handling exceptions, the preset priority level, and the current number of idle threads specifically comprises:
the exception handling time interval is determined by the following formula:
wherein,for exception handling time interval, +.>To deal with the number of exceptions, +.>For correction factor +.>For the number of data sources corresponding to the abnormal business processing sub-flow,/-for the abnormal business processing sub-flow>For the corresponding input data quantity in the ith data source of the abnormal business processing sub-flow,/the sub-flow is processed>For preset priority level, ++>Is the current number of idle threads.
7. The method of claim 1, wherein before the sending the processing log and the call log to an operation and maintenance person, the method further comprises:
determining a service data set corresponding to the abnormal service processing sub-flow;
and determining user private data in the service data set, and encrypting the user private data.
8. The method according to claim 1, wherein after the processing the target data through the target service invocation interface, the method specifically comprises:
determining that the processing result of the target service on the target data is normal processing;
and recording the normal processing result of the abnormal business processing sub-flow.
9. An automated processing apparatus for multi-source data services, comprising:
at least one processor; and a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform:
determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively;
receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow;
selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface;
if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface;
and sending the processing log and the call record to an operation and maintenance personnel.
10. A non-transitory computer storage medium storing computer-executable instructions, the computer-executable instructions configured to:
determining a plurality of data sources corresponding to the target multi-source data service and service calling interfaces corresponding to the service processing sub-flows respectively;
receiving processing exception information from a business processing sub-flow, and determining a target data source and a target business call interface corresponding to the abnormal business processing sub-flow;
selecting target data in the target data source again based on a preset time interval, and processing the target data through the target service calling interface;
if the abnormal times of the abnormal business processing sub-flows are higher than a preset threshold, acquiring processing logs corresponding to the business processing sub-flows and call records of the business call interface;
and sending the processing log and the call record to an operation and maintenance personnel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410008109.3A CN117522349B (en) | 2024-01-04 | 2024-01-04 | Automatic processing method, equipment and medium for multi-source data service |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410008109.3A CN117522349B (en) | 2024-01-04 | 2024-01-04 | Automatic processing method, equipment and medium for multi-source data service |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117522349A true CN117522349A (en) | 2024-02-06 |
| CN117522349B CN117522349B (en) | 2024-03-29 |
Family
ID=89766791
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410008109.3A Active CN117522349B (en) | 2024-01-04 | 2024-01-04 | Automatic processing method, equipment and medium for multi-source data service |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117522349B (en) |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070061882A1 (en) * | 2005-09-13 | 2007-03-15 | Honeywell International Inc. | Instance based learning framework for effective behavior profiling and anomaly intrusion detection |
| US20150379062A1 (en) * | 2014-06-26 | 2015-12-31 | Amazon Technologies, Inc. | Multi-database log with multi-item transaction support |
| CN108595307A (en) * | 2018-05-03 | 2018-09-28 | 广州供电局有限公司 | A kind of automatic self-healing method based on IT O＆Ms |
| CN109992475A (en) * | 2019-03-20 | 2019-07-09 | 网宿科技股份有限公司 | A kind of processing method of log, server and storage medium |
| CN111459754A (en) * | 2020-03-26 | 2020-07-28 | 平安普惠企业管理有限公司 | Abnormal task processing method, device, medium and electronic equipment |
| CN111475515A (en) * | 2020-03-05 | 2020-07-31 | 深圳壹账通智能科技有限公司 | Compensation management method and device for failed task, computer equipment and storage medium |
| CN111899105A (en) * | 2020-08-04 | 2020-11-06 | 深圳乐信软件技术有限公司 | System exception handling method and device, computer equipment and storage medium |
| US20210216914A1 (en) * | 2018-08-03 | 2021-07-15 | Sony Corporation | Information processing device, information processing method, and information processing program |
| CN113238922A (en) * | 2021-06-02 | 2021-08-10 | 南京领行科技股份有限公司 | Log analysis method and device, electronic equipment and medium |
| CN113448712A (en) * | 2021-07-12 | 2021-09-28 | 中国银行股份有限公司 | Task scheduling execution method and device |
| CN114924904A (en) * | 2022-06-22 | 2022-08-19 | 平安科技（深圳）有限公司 | Failed task retry method and device, electronic equipment and storage medium |
| CN115373868A (en) * | 2021-07-18 | 2022-11-22 | 周志燕 | Intelligent data processing system and method |
| CN116383083A (en) * | 2023-04-23 | 2023-07-04 | 中航信移动科技有限公司 | Multi-interface connection-based abnormal data source determining method and storage medium |
| CN116560896A (en) * | 2023-07-11 | 2023-08-08 | 智慧眼科技股份有限公司 | Abnormality compensation method, device, equipment and storage medium |
| WO2023168926A1 (en) * | 2022-03-09 | 2023-09-14 | 青岛海尔科技有限公司 | Method and apparatus for determining software anomaly, storage medium, and electronic apparatus |
-
2024
- 2024-01-04 CN CN202410008109.3A patent/CN117522349B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070061882A1 (en) * | 2005-09-13 | 2007-03-15 | Honeywell International Inc. | Instance based learning framework for effective behavior profiling and anomaly intrusion detection |
| US20150379062A1 (en) * | 2014-06-26 | 2015-12-31 | Amazon Technologies, Inc. | Multi-database log with multi-item transaction support |
| CN108595307A (en) * | 2018-05-03 | 2018-09-28 | 广州供电局有限公司 | A kind of automatic self-healing method based on IT O＆Ms |
| US20210216914A1 (en) * | 2018-08-03 | 2021-07-15 | Sony Corporation | Information processing device, information processing method, and information processing program |
| CN109992475A (en) * | 2019-03-20 | 2019-07-09 | 网宿科技股份有限公司 | A kind of processing method of log, server and storage medium |
| CN111475515A (en) * | 2020-03-05 | 2020-07-31 | 深圳壹账通智能科技有限公司 | Compensation management method and device for failed task, computer equipment and storage medium |
| CN111459754A (en) * | 2020-03-26 | 2020-07-28 | 平安普惠企业管理有限公司 | Abnormal task processing method, device, medium and electronic equipment |
| CN111899105A (en) * | 2020-08-04 | 2020-11-06 | 深圳乐信软件技术有限公司 | System exception handling method and device, computer equipment and storage medium |
| CN113238922A (en) * | 2021-06-02 | 2021-08-10 | 南京领行科技股份有限公司 | Log analysis method and device, electronic equipment and medium |
| CN113448712A (en) * | 2021-07-12 | 2021-09-28 | 中国银行股份有限公司 | Task scheduling execution method and device |
| CN115373868A (en) * | 2021-07-18 | 2022-11-22 | 周志燕 | Intelligent data processing system and method |
| WO2023168926A1 (en) * | 2022-03-09 | 2023-09-14 | 青岛海尔科技有限公司 | Method and apparatus for determining software anomaly, storage medium, and electronic apparatus |
| CN114924904A (en) * | 2022-06-22 | 2022-08-19 | 平安科技（深圳）有限公司 | Failed task retry method and device, electronic equipment and storage medium |
| CN116383083A (en) * | 2023-04-23 | 2023-07-04 | 中航信移动科技有限公司 | Multi-interface connection-based abnormal data source determining method and storage medium |
| CN116560896A (en) * | 2023-07-11 | 2023-08-08 | 智慧眼科技股份有限公司 | Abnormality compensation method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| STEPHEN RANSHOUS等: ""Anomaly detection in dynamic networks: a survey"", 《WIRES COMPUTATIONAL STATISTICS》, vol. 7, 30 June 2015 (2015-06-30), pages 223 - 247, XP055560851, DOI: 10.1002/wics.1347 * |
| 贾宝刚;: "一种基于数据挖掘的网络入侵检测系统", 科技资讯, no. 21, 23 July 2007 (2007-07-23), pages 116 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117522349B (en) | 2024-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108647105B (en) | Idempotent control method, device and system in system switching process | |
| CN112486719A (en) | Method and equipment for RPC interface call failure processing | |
| CN117522349B (en) | Automatic processing method, equipment and medium for multi-source data service | |
| CN111125115A (en) | Data storage method, device and medium based on block chain | |
| CN110858166A (en) | Application exception processing method and device, storage medium and processor | |
| CN113411209A (en) | Distributed password service full-link detection system and method | |
| CN113225191B (en) | Generation method and device of consensus node, storage medium and processor | |
| CN112749142B (en) | Handle management method and system | |
| CN116266328A (en) | Account data checking method and device and electronic equipment | |
| CN114691395A (en) | Fault processing method and device, electronic equipment and storage medium | |
| CN113468446A (en) | Method, system and equipment for supporting identification of third-party two-dimensional code data | |
| CN112261035A (en) | Information management method based on block chain, prevention and control center node and rework platform | |
| CN111444201A (en) | Block chain-based security trading process optimization method, equipment and medium | |
| CN113742664B (en) | Monitoring and auditing method, equipment and system | |
| CN117522601B (en) | Nuclear insurance service execution method, equipment and medium based on interface unified management | |
| CN108710573A (en) | It is a kind of that test method, device, storage medium and terminal a little are buried based on daily record | |
| CN117453251A (en) | Processing program updating method, device, equipment and storage medium | |
| CN117170974A (en) | Log acquisition method, device and equipment | |
| CN117439797A (en) | Connection management method and device of cipher machine, storage medium and electronic equipment | |
| CN115061845A (en) | Method and device for confirming abnormal node in backup process and storage medium | |
| CN117768155A (en) | Access request processing method and device, storage medium and electronic equipment | |
| CN117389573A (en) | Interface calling method, device and medium based on intelligent interface integrated system | |
| CN117271474A (en) | Data migration method and device, storage medium and electronic equipment | |
| CN116521674A (en) | Data processing method, device, equipment and storage medium | |
| CN117033476A (en) | Data aggregation method, device and medium for region |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |