CN117609352A - Data processing method, device, equipment and storage medium - Google Patents

Data processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN117609352A
CN117609352A CN202311639285.9A CN202311639285A CN117609352A CN 117609352 A CN117609352 A CN 117609352A CN 202311639285 A CN202311639285 A CN 202311639285A CN 117609352 A CN117609352 A CN 117609352A
Authority
CN
China
Prior art keywords
data
format
service
target
service data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311639285.9A
Other languages
Chinese (zh)
Inventor
蒙永明
李晓轩
戴树江
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Agricultural Bank of China
Original Assignee
Agricultural Bank of China
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Agricultural Bank of China filed Critical Agricultural Bank of China
Priority to CN202311639285.9A priority Critical patent/CN117609352A/en
Publication of CN117609352A publication Critical patent/CN117609352A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2453—Query optimisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention provides a data processing method, a device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: responding to the data storage requirement of the first service, and acquiring the service data to be stored in the first format generated by the first service; converting the business data to be stored from a first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode; responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage; and converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service. According to the technical scheme provided by the invention, the intermediate results are stored in the form of the target format, so that the intermediate results of all batch processing links are uniformly processed, the batch processing flow is simpler and more efficient, and meanwhile, the intermediate results are convenient to store and use, so that the use efficiency of the intermediate results is improved.

Description

Data processing method, device, equipment and storage medium
Technical Field
Embodiments of the present invention relate to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
In enterprise applications, there are a number of background batch jobs in addition to online transactions. Batch processing operation is a series of automatic processing workflow sets, and data needs to be shared among batch processing links, namely, data after a follow-up link needs to be processed by using a preceding link, and most of the data is a temporary intermediate result. Because the intermediate result needs to be shared by a plurality of batch processing links, the enterprise application system must consider a mechanism for transmitting and sharing the intermediate result among the batch processing links, including a storage mode, a reading mode, a destroying mode and the like of the intermediate result.
In the process of implementing the present invention, the inventor finds that, due to the great difference between the batch logic and the technical routes between different systems, an intermediate result processing mechanism (such as a storage mode, a reading mode and a destroying mode) which is specific to each batch processing link needs to be developed for each batch processing link, so that the use efficiency of the intermediate result is reduced, and the availability of the intermediate result is poor.
Disclosure of Invention
The embodiment of the invention provides a data processing method, a device, equipment and a storage medium, which store intermediate results in a target format form, do not need to develop an intermediate result processing mechanism special for each batch processing link, realize unified processing of the intermediate results of each batch processing link, enable batch processing flow to be more concise and efficient, and simultaneously enable the intermediate results to be convenient to store and use, further improve the availability of the intermediate results, and further improve the processing efficiency of batch processing and the use efficiency of the intermediate results.
In a first aspect, a data processing method provided by an embodiment of the present invention includes:
responding to the data storage requirement of the first service, and acquiring the service data to be stored in the first format generated by the first service;
converting the business data to be stored from a first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode;
responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage;
and converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service.
In a second aspect, a data processing apparatus provided in an embodiment of the present invention includes:
the first acquisition module is used for responding to the data storage requirement of the first service and acquiring the service data to be stored in the first format generated by the first service;
the storage module is used for converting the business data to be stored from the first format into the target format to obtain intermediate business data, and storing the intermediate business data into the target storage according to the structured storage mode;
the second acquisition module is used for responding to the data acquisition requirement of the second service and acquiring intermediate service data from the target storage;
And the feedback module is used for converting the intermediate service data from the target format to the second format, obtaining the target service data and feeding back the target service data to the second service.
In a third aspect, an electronic device provided in an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements a data processing method according to any of the embodiments of the present invention when executing the program.
In a fourth aspect, embodiments of the present invention provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data processing method according to any of the embodiments of the present invention.
According to the scheme provided by the embodiment of the invention, the data storage requirement of the first service can be responded, and the service data to be stored in the first format generated by the first service can be obtained; converting the business data to be stored from a first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode; responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage; and converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service. In the above technical solution, the data storage requirement of the first service can be responded, the service data to be stored generated by the first service is obtained, then the service data to be stored with the data format of the first format is assembled according to the target format to obtain the intermediate service data with the data format of the target format, for example, the information such as the service identifier, the execution time, the data name, the data content, the storage time and the like of the service data to be stored can be assembled, and the intermediate service data is used as a record and stored to the target storage in a structured storage mode, so that the storage function of the intermediate result is realized; then, responding and analyzing the data acquisition requirement of the second service to obtain the data name of the intermediate service data which needs to be acquired by the second service, and acquiring the intermediate service data corresponding to the data name from the target storage, for example, searching the target storage according to the data name to obtain the intermediate service data corresponding to the data name, then analyzing the intermediate service data which is in the target format according to the second format of the second service to obtain the target service data which is in the second format, and transmitting the target service data to the second service, thereby realizing the reading function of the intermediate result; in addition, the intermediate results are stored in the form of the target format, an intermediate result processing mechanism special for each batch processing link is not required to be developed, the intermediate results of the batch processing links are uniformly processed, the batch processing flow is more concise and efficient, the intermediate results are convenient to store and use, the availability of the intermediate results is further improved, and therefore the processing efficiency of batch processing and the use efficiency of the intermediate results are improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and should not be considered as limiting the scope, and that other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of another flow chart of a data processing method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention, where the data processing method according to the embodiment of the present invention is applicable to a scenario where intermediate results need to be shared among batch processing links, the method may be performed by a data processing apparatus according to an embodiment of the present invention, and the apparatus may be implemented in a software and/or hardware manner. In a specific embodiment, the apparatus may be integrated in an electronic device, in particular a batch process deployed in the electronic device, e.g. the electronic device may be a computer, a server, etc. The execution body for executing the method can be a batch processing program of the electronic equipment, referring to fig. 1, the method specifically comprises the following steps:
S110, responding to the data storage requirement of the first service, and acquiring the service data to be stored in the first format generated by the first service.
In banking, most operations need to rely on intermediate results that are passed from one link to another and used in subsequent processing. The following are several example links and related intermediate results in banking: 1. in the process of opening an account, a unique customer account number can be generated as an intermediate result, and the account number can be used for subsequent deposit, withdrawal, transfer and other operations; 2. a deposit, in which the intermediate result may be deposit amount, deposit date, and deposit receipt/voucher information, which will be used for subsequent account balance updates and transaction records; 3. transferring, in the transferring operation, the intermediate result may be information such as a transfer account, a transfer amount, and a transfer date, which are used to verify the account balance and update the transfer record; 4. withdrawal, in which the intermediate result may be withdrawal amount, withdrawal date, and withdrawal receipt, etc., which will be used to update the account balance and generate a transaction record; 5. the loan approval, in the loan approval process, the intermediate result can be information such as approval state, loan amount, loan interest rate and the like, and the information can determine whether to approve the loan and generate subsequent loan contracts; 6. in reconciliation, the intermediate results may be account statement, transaction details, and difference reports, etc. that are used to check account balances, identify problem transactions, and correct errors.
The first business may be any business in the banking business, for example, the first business may be a customer account opening, deposit, transfer, withdrawal, loan approval, account checking, or the like, and optionally, the first business may be 1 or more, which is not particularly limited in the embodiment of the present invention.
The data storage requirement is a request generated by a first service for requesting the batch program to store service data to be stored.
The service data to be stored is the service data which is generated by the first service and needs to be stored during the processing process of the first service or after the processing is completed. The first format is a data format of service data to be stored generated by the first service.
Specifically, at any time that the first business wants to store data, the first business can generate a data storage requirement and send the data storage requirement to the batch process.
Then, the batch processing program can receive the data storage requirement of the first service and respond to the data storage requirement of the first service, then acquire the service data to be stored generated by the first service, for example, after receiving the data storage requirement of the first service, the batch processing program generates a data acquisition command and sends the data acquisition command to the first service, then the first service receives the data acquisition command and sends the service data to be stored to the batch processing program; or the first service simultaneously sends the data storage requirement and the service data to be stored which need to be stored to the batch processing program, and then the batch processing program simultaneously receives the data storage requirement and the service data to be stored of the first service.
S120, converting the business data to be stored from the first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode.
The target format is a preset data format and is used for representing the format of data when the data is stored in the target storage. By way of example, the target format may be a JS key-pair data (JavaScript Object Notation, JSON) format, an extensible markup language (Extensible Markup Language, XML) format, a Comma-Separated Values (CSV) file format, another markup language (YAML Ain't Markup Language, YAML) format, or a binary-serialized document format (Binary Serialized Document Format, BSON), etc., as embodiments of the present invention are not limited in this regard. The target storage may be a database, a data file, or a memory, etc. for storing data, which is not specifically limited in the embodiment of the present invention.
The intermediate service data is data obtained after the service data to be stored, which is in the first format, is assembled according to the target format. Optionally, the intermediate service data may include information such as a service identifier, an execution time, a data name, a data content, and a storage time to which the service data to be stored belongs, which is not particularly limited in the embodiment of the present invention.
The structured storage mode is to store data in the form of structured data. Structured data is data having a fixed format and rules, typically presented in the form of tables, tree structures, or relational models, primarily stored and managed by relational databases.
Specifically, after obtaining the service data to be stored generated by the first service, the batch processing program may process the format of the service data to be stored, that is, convert the first format of the service data to be stored into a preset target format, obtain intermediate service data with a data format being the target format, and then store the intermediate service data with the data format being the target format into the target storage according to the structured storage mode. For example, after obtaining the service data to be stored with the data format being the first format, the service data to be stored may be assembled according to the target format, that is, information such as service identifier, execution time, data name, data content, and storage time to which the service data to be stored belongs is assembled, so as to obtain intermediate service data with the data format being the target format, and then the intermediate service data is used as a record and stored in the target storage in a structured storage manner. The service identifier is a first service identifier, is a unique identifier of a service, and is used for representing the service for generating the service data to be stored; the execution time is the execution time of the first service; the data name is the data name of the service data to be stored and is the unique identification of the service data to be stored; the data content is the data content of the service data to be stored; the storing time is the time for storing the business data to be stored.
Illustratively, when the data name of the service data to be stored is detail_attr, the data content of the service data to be stored is 300, the first service identifier is 1000, and the target format is JSON, the intermediate service data obtained after the service data to be stored is assembled according to the target format is { "detail_attr" { "service identifier": [ "1000" ], "execution time": [ "20230216" ], "data content": 300"]," deposit time ": ["20230216101010"] }.
S130, responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage.
The second business may be any business in the banking business, and exemplary, the second business may be a customer account opening, deposit, transfer, withdrawal, loan approval, account checking, or the like, and optionally, the second business may be 1 or more, which is not particularly limited in the embodiment of the present invention.
It should be noted that the first service and the second service are different links of the service, and the second service is located after the first service.
The data acquisition requirement is a request generated by the second service to indicate that intermediate service data is to be acquired from the target storage.
Specifically, at any time when the second service wants to acquire data generated by the first service, the second service may generate a data acquisition requirement and send the data acquisition requirement to the batch process.
After the intermediate service data of the first service is stored in the target storage, responding to the data acquisition requirement of the second service, analyzing the data acquisition requirement to obtain the data name of the intermediate service data required to be acquired by the second service, and searching the target storage according to the data name to obtain the intermediate service data corresponding to the data name.
S140, converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service.
The second format is a data format corresponding to the second service. The target service data is data obtained after the second service analyzes the intermediate service data with the data format being the target format according to the second format.
Specifically, after the batch processing program obtains the intermediate service data from the target storage, the batch processing program may obtain a second format corresponding to the second service, then parse the intermediate service data with the data format being the target format according to the second format, so as to convert the target format of the intermediate service data into the second format corresponding to the second service, obtain the target service data with the data format being the second format, and finally send the target service data to the second service.
According to the scheme provided by the embodiment of the invention, the data storage requirement of the first service can be responded, and the service data to be stored in the first format generated by the first service can be obtained; converting the business data to be stored from a first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode; responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage; and converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service. In the above technical solution, the data storage requirement of the first service can be responded, the service data to be stored generated by the first service is obtained, then the service data to be stored with the data format of the first format is assembled according to the target format to obtain the intermediate service data with the data format of the target format, for example, the information such as the service identifier, the execution time, the data name, the data content, the storage time and the like of the service data to be stored can be assembled, and the intermediate service data is used as a record and stored to the target storage in a structured storage mode, so that the storage function of the intermediate result is realized; then, responding and analyzing the data acquisition requirement of the second service to obtain the data name of the intermediate service data which needs to be acquired by the second service, and acquiring the intermediate service data corresponding to the data name from the target storage, for example, searching the target storage according to the data name to obtain the intermediate service data corresponding to the data name, then analyzing the intermediate service data which is in the target format according to the second format of the second service to obtain the target service data which is in the second format, and transmitting the target service data to the second service, thereby realizing the reading function of the intermediate result; in addition, the intermediate results are stored in the form of the target format, an intermediate result processing mechanism special for each batch processing link is not required to be developed, the intermediate results of the batch processing links are uniformly processed, the batch processing flow is more concise and efficient, the intermediate results are convenient to store and use, the availability of the intermediate results is further improved, and therefore the processing efficiency of batch processing and the use efficiency of the intermediate results are improved.
Fig. 2 is another flow chart of a data processing method according to an embodiment of the present invention, where the data processing method according to the embodiment of the present invention is applicable to a scenario where intermediate results need to be shared among batch processing links. The data processing method of the embodiment of the present invention, which is optimized based on the above embodiment, as shown in fig. 2, may include the following steps:
s201, responding to the data storage requirement of the first service, and acquiring the service data to be stored in the first format generated by the first service.
S202, determining whether the first format is the same as the target format.
Specifically, after obtaining the service data to be stored in the first format generated by the first service, the batch processing program may obtain a preset target format, and then compare the first format with the target format to determine whether the first format is the same as the target format, that is: if the first format is the same as the target format, it indicates that there is no need to convert the data format of the service data to be stored, and S203 is executed at this time; if the first format is different from the target format, it indicates that the data format of the service data to be stored needs to be converted, and S204 is performed.
And S203, when the first format is the same as the target format, determining the service data to be stored as intermediate service data.
Specifically, when the first format is the same as the target format, it indicates that there is no need to convert the data format of the service data to be stored. At this time, the batch process may directly determine the service data to be stored as intermediate service data. The intermediate service data comprises service identification, execution time, data name, data content, storage time and other information of the service data to be stored.
After that, S205 is performed to store the intermediate service data.
S204, when the first format is different from the target format, converting the service data to be stored from the first format into the target format to obtain intermediate service data.
Specifically, when the first format is different from the target format, it indicates that the data format of the service data to be stored needs to be converted. At this time, the batch processing program may assemble the service data to be stored with the data format being the first format according to the target format, that is, assemble the information such as the service identifier, the execution time, the data name, the data content, the storage time, etc. to which the service data to be stored belongs, to obtain the intermediate service data with the data format being the target format. After that, S205 is performed to store the intermediate service data.
S205, calling a preset storage interface of the target storage to store the intermediate service data into the target storage according to a structured storage mode.
The preset storage interface is a pre-packaged storage interface and is used for storing data into a database, a data file or a memory, and in the embodiment of the invention, the batch processing program can directly call the preset storage interface, namely, intermediate service data can be stored into a target storage. Optionally, the preset storage interface may support multiple batch programs to be invoked concurrently.
Optionally, before the batch processing program responds to the data storage requirement of each service, the storage interface of the target storage may be pre-packaged, that is, the intermediate service data with the data format being the target format may be used as a record, and stored in the target storage in a structured storage manner.
Specifically, after the batch processing program obtains the intermediate service data, the batch processing program may call a preset storage interface of the target storage, and then store the intermediate service data with the data format being the target format to the target storage in a structured storage mode by using the preset storage interface.
S206, setting access authority for the intermediate service data according to the indication of the first service.
Wherein the access right is a mechanism for limiting access to certain information items or certain controls according to the identity of the user and the membership of the user in various predefined groups, in the embodiment of the present invention, the access right is used to indicate that a service in each batch processing link can access certain intermediate service data stored in the target storage, that is, only a service having the access right of the intermediate service data can access the intermediate service data.
In particular, the first service may send the accessible service of the service data to be stored to the batch process after sending the data storage requirement to the batch process. The batch process may then receive the accessible service of the service data to be stored sent by the first service. Wherein an accessible service is used to represent a service that can access the service data to be stored.
The intermediate service data is to-be-stored service data or is obtained after format conversion of the to-be-stored service data, so that the accessible service of the to-be-stored service data is the accessible service of the intermediate service data.
After the batch processing program stores the intermediate service data in the target storage, an access right can be set for the intermediate service data according to the received accessible service of the intermediate service data, for example, a service identification list capable of accessing the intermediate service data is set, the access right of the intermediate service data is obtained, and the access right is stored.
Optionally, in one implementation, after setting the access right for the intermediate service data according to the indication of the first service, the batch processing program may generate the access prompt information, and then send the access prompt information to the service having the access right for the intermediate service data, so as to prompt the service having the access right to timely acquire the intermediate service data from the target storage.
Alternatively, in another implementation, after setting the access right for the intermediate service data according to the indication of the first service, the batch processing program may determine, when receiving the data acquisition requirement of the service, whether to allow the service to acquire the intermediate service data from the target storage according to the access right of the intermediate service data.
S207, determining whether the second service has the access right of the intermediate service data.
Specifically, at any time when the second service wants to acquire data generated by the first service, the second service may generate a data acquisition requirement and send the data acquisition requirement to the batch process.
After setting the access right for the intermediate service data, the batch processing program can determine whether the second service has the access right for the intermediate service data when receiving the data acquisition requirement of the second service so as to judge whether the second service can access the intermediate service data of the first service. Specifically, the service identifier of the second service may be compared with the service identifier in the access right to determine whether the service identifier of the second service is in the access right, if the service identifier of the second service is in the access right, it indicates that the second service has the access right of the intermediate service data, and at this time, S208 is executed to obtain the intermediate service data; if the service identifier of the second service is not in the access right, it indicates that the second service does not have the access right of the intermediate service data, and S215 is executed at this time to generate the prompt information.
S208, when the second service has the access right of the intermediate service data, the preset access number interface of the target storage is called in response to the data acquisition requirement of the second service so as to acquire the intermediate service data from the target storage.
The preset access interface is a pre-packaged access interface and is used for acquiring data from a database, a data file or a memory, and in the embodiment of the invention, the batch processing program can directly call the preset access interface, namely, intermediate service data can be acquired from a target storage. Optionally, the preset fetch interface may support multiple batch concurrent calls.
Optionally, before the batch processing program responds to the data acquisition requirement of each service, the access interface of the target storage may be pre-packaged, that is, the target storage may be searched according to the data name, so as to obtain the intermediate service data corresponding to the data name.
Specifically, when the second service has the access right of the intermediate service data, the batch processing program may respond to the data acquisition requirement of the second service, analyze the data acquisition requirement to obtain the data name of the intermediate service data to be acquired by the second service, and then call the preset access interface of the target storage according to the data name to search the target storage to obtain the intermediate service data corresponding to the data name.
S209, determining whether the second format is the same as the target format.
Specifically, after obtaining the intermediate service data from the target storage, the batch processing program may acquire a second format corresponding to the second service, and then compare the second format with the target format to determine whether the second format is the same as the target format, that is: if it is determined that the second format is the same as the target format, it indicates that the data format of the intermediate service data does not need to be modified, and S210 is performed at this time; if it is determined that the second format is different from the target format, it indicates that the data format of the intermediate service data needs to be modified, at which point S211 is performed.
And S210, when the second format is the same as the target format, determining the intermediate service data as the target service data.
In particular, when the second format is the same as the target format, it indicates that no modification of the data format of the intermediate service data is required. At this time, the batch process may directly determine the intermediate service data as the target service data. After that, S212 is performed to transmit the target service data to the second service.
S211, when the second format is different from the target format, converting the intermediate service data from the target format to the second format to obtain the target service data.
Specifically, when the second format is different from the target format, it indicates that the data format of the intermediate service data needs to be modified. At this time, the batch processing program may parse the intermediate service data with the data format being the target format according to the second format, so as to convert the target format of the intermediate service data into the second format corresponding to the second service, and obtain the target service data with the data format being the second format. After that, S212 is performed to transmit the target service data to the second service.
S212, feeding back target service data to the second service.
Specifically, after obtaining the target service data, the batch processing program may send the target service data that has been obtained to the second service.
S213, determining whether a preset deleting condition is reached.
The preset deletion condition is a preset condition for deleting the intermediate service data stored in the target storage, and the user can adjust and set the preset deletion condition according to the actual use requirement, which is not particularly limited in the embodiment of the present invention.
Optionally, the preset deletion condition may be that the plurality of second services all acquire data successfully, the storage duration of the intermediate service data exceeds a preset duration threshold, the intermediate service data fails or the remaining storage space of the target storage is zero or smaller than a preset storage threshold. The storage duration is the difference between the current time and the storage time of the intermediate service data.
The preset duration threshold is a preset duration threshold and is used for determining whether the storage duration of the intermediate service data exceeds the duration of the target storage which can be stored continuously, and a user can adjust and set the preset duration threshold according to actual use requirements, so that the embodiment of the invention is not limited in detail; when the storage time length is greater than a preset time length threshold value, indicating that the target storage can not continuously store the intermediate service data; and when the storage duration is less than or equal to a preset duration threshold value, indicating that the target storage can continue to store the intermediate service data.
The preset storage threshold is a preset storage threshold, and is used for indicating whether the residual storage space of the target storage can continuously store new intermediate service data, and the user can adjust and set the preset storage threshold according to actual use requirements, so that the embodiment of the invention is not particularly limited; when the residual storage space of the target storage is smaller than a preset storage threshold value, indicating that the target storage cannot continuously store new intermediate service data; and when the remaining storage space of the target storage is greater than or equal to a preset storage threshold value, indicating that the target storage can continue to store new intermediate service data.
Specifically, after feeding back the target service data to the second service, the batch processing program may determine whether the intermediate service data acquired by the second service reaches a preset deletion condition, that is: if the intermediate service data acquired by the second service reach the preset deletion condition, indicating that the plurality of second services have acquired the intermediate service data, wherein the storage duration of the intermediate service data exceeds a preset duration threshold, the intermediate service data fails or the residual storage space of the target storage is zero or less than the preset storage threshold, and executing S214 at the moment to delete the intermediate service data; if the intermediate service data acquired by the second service does not reach the preset deletion condition, it indicates that the plurality of second services do not acquire the intermediate service data, the storage duration of the intermediate service data does not exceed the preset duration threshold, the intermediate service data is not invalid or the remaining storage space of the target storage is greater than or equal to the preset storage threshold, at this time, the intermediate service data does not need to be deleted, and S207 is executed to continue responding to the data acquisition requirement of the second service.
And S214, deleting the intermediate service data stored in the target storage when the preset deleting condition is reached.
Specifically, when the preset deletion condition is reached, the batch processing program may call the preset deletion interface of the target storage to delete the intermediate service data stored in the target storage. The preset deletion interface is a pre-packaged deletion interface and is used for deleting data from a database, a data file or a memory, and in the embodiment of the invention, the batch processing program can directly call the preset deletion interface, namely, intermediate service data stored in the target storage can be deleted. Optionally, the preset deletion interface may support multiple batch concurrent calls.
Optionally, before the batch processing program deletes the intermediate service data stored in the target storage, the deletion interface may be pre-packaged, that is, the target storage may be searched according to the data name, so as to obtain the intermediate service data corresponding to the data name, and then the intermediate service data is deleted from the target storage.
S215, when the second service does not have the access right of the intermediate service data, generating prompt information.
Specifically, when the second service does not have the access right of the intermediate service data, it is indicated that the second service does not have the access right of the intermediate service data. At this time, the batch processing program generates a prompt message, and sends the prompt message to the second service to notify that the second service does not have the access right of the data, and cannot acquire the data.
According to the scheme provided by the embodiment of the invention, the data storage requirement of the first service can be responded, and the service data to be stored in the first format generated by the first service can be obtained; determining whether the first format is the same as the target format; when the first format is the same as the target format, determining the service data to be stored as intermediate service data; when the first format is different from the target format, converting the service data to be stored from the first format into the target format to obtain intermediate service data; calling a preset storage interface of the target storage to store the intermediate service data into the target storage according to a structured storage mode; setting access rights for the intermediate service data according to the indication of the first service; determining whether the second service has access rights to the intermediate service data; when the second service has the access right of the intermediate service data, responding to the data acquisition requirement of the second service, and calling a preset access interface of the target storage to acquire the intermediate service data from the target storage; determining whether the second format is the same as the target format; when the second format is the same as the target format, determining the intermediate service data as target service data; when the second format is different from the target format, converting the intermediate service data from the target format to the second format to obtain target service data; feeding back target service data to the second service; determining whether a preset deletion condition is reached; when a preset deleting condition is reached, deleting the intermediate service data stored in the target storage; and generating prompt information when the second service does not have the access right of the intermediate service data. According to the technical scheme, the intermediate results are stored in the form of the target format, an intermediate result processing mechanism special for each batch processing link is not required to be developed, the intermediate results of all batch processing links are uniformly processed, the batch processing flow is simpler and more efficient, meanwhile, the intermediate results are convenient to store and use, the availability of the intermediate results is further improved, and therefore the processing efficiency of batch processing and the use efficiency of the intermediate results are improved; further, when the first format is determined to be different from the target format or the second format is determined to be different from the target format, format conversion is performed on the service data to be stored or the intermediate service data, so that the data storage and data reading rates are improved, and when the first format is determined to be the same as the target format or the second format is determined to be the same as the target format, format conversion is not performed on the service data to be stored or the intermediate service data, so that processing resources are saved; the storage and the acquisition of the intermediate result are realized by calling the common preset storage interface and the preset fetch interface concurrently, and the data storage and data reading rates are further improved, so that the processing efficiency of batch processing and the use efficiency of the intermediate result are further improved; when the second service is determined to have the access right of the intermediate service data, the intermediate service data is acquired from the target storage, so that the data safety is ensured; when the preset deleting condition is determined to be reached, deleting the intermediate service data stored in the target storage, so that the destroying function of the intermediate result is realized, the intermediate result can be processed in time, the storage resource is saved, and the utilization rate of the storage resource is further improved.
Fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, where the apparatus is adapted to perform a data processing method according to an embodiment of the present invention, and as shown in fig. 3, the data processing apparatus may specifically include:
a first obtaining module 310, configured to obtain, in response to a data storage requirement of a first service, service data to be stored in a first format generated by the first service;
the storage module 320 is configured to convert the service data to be stored from the first format to the target format, obtain intermediate service data, and store the intermediate service data in the target storage according to the structured storage manner;
a second obtaining module 330, configured to obtain intermediate service data from the target storage in response to a data obtaining requirement of the second service;
and the feedback module 340 is configured to convert the intermediate service data from the target format to the second format, obtain the target service data, and feed back the target service data to the second service.
In one embodiment, the data processing apparatus further includes a format determining module, where the format determining module is specifically configured to:
before converting the business data to be stored from the first format to the target format, determining whether the first format is the same as the target format, and triggering to execute the conversion of the business data to be stored from the first format to the target format when the first format is different from the target format;
Before converting the intermediate business data from the target format to the second format, determining whether the second format is the same as the target format, and triggering the execution of the conversion of the intermediate business data from the target format to the second format when the second format is different from the target format.
In one embodiment, the storage module 320 stores the intermediate service data in the target storage according to a structured storage manner, including:
calling a preset storage interface of the target storage to store the intermediate service data into the target storage according to a structured storage mode;
the second obtaining module 330 obtains intermediate service data from the target storage, including: and calling a preset fetch interface of the target storage to acquire the intermediate service data from the target storage.
In one embodiment, the data processing apparatus further includes a deletion module, where the deletion module is specifically configured to:
and deleting the intermediate service data stored in the target storage.
In one embodiment, the data processing apparatus further includes a deletion determination module, where the deletion determination module is specifically configured to:
before deleting the intermediate service data stored in the target storage, determining whether a preset deleting condition is reached;
and triggering and executing the intermediate business data stored in the deletion target storage when the preset deletion condition is reached.
In one embodiment, the data processing apparatus further includes a rights setting module, where the rights setting module is specifically configured to:
after the intermediate service data is stored in the target storage according to the structured storage mode, the access authority is set for the intermediate service data according to the indication of the first service.
In one embodiment, the data processing apparatus further includes a permission judging module, where the permission judging module is specifically configured to:
before acquiring the intermediate service data from the target storage, determining whether the second service has access rights to the intermediate service data;
and triggering execution of acquiring the intermediate service data from the target storage when the second service has the access right of the intermediate service data.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above. The specific working process of the functional module described above may refer to the corresponding process in the foregoing method embodiment, and will not be described herein.
The data processing device provided by the embodiment of the invention can be applied to the data processing method provided by any embodiment, and has corresponding functions and beneficial effects.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. Fig. 4 shows a block diagram of an exemplary electronic device 11 suitable for use in implementing embodiments of the invention. The electronic device 11 shown in fig. 4 is only an example, and should not impose any limitation on the function and the range of use of the present embodiment.
As shown in fig. 4, the electronic device 11 is in the form of a general purpose computing electronic device. Components of the electronic device 11 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, a bus 18 that connects the various system components, including the system memory 28 and the processing units 16.
Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.
The electronic device 11 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by electronic device 11 and includes both volatile and non-volatile media, removable and non-removable media.
The system memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and/or cache memory 32. The electronic device 11 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 34 may be used to read from or write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, commonly referred to as a "hard disk drive"). Although not shown in fig. 4, a magnetic disk drive for reading from and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from or writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In such cases, each drive may be coupled to bus 18 through one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to carry out the functions of the embodiments of the invention.
A program/utility 40 having a set (at least one) of program modules 42 may be stored in, for example, system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment. Program modules 42 generally perform the functions and/or methods of the embodiments described herein.
The electronic device 11 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the electronic device 11, and/or any devices (e.g., network card, modem, etc.) that enable the electronic device 11 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 22. Also, the electronic device 11 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through a network adapter 20.
As shown in fig. 4, the network adapter 20 communicates with other modules of the electronic device 11 via the bus 18. It should be appreciated that although not shown in fig. 4, other hardware and/or software modules may be used in connection with the electronic device 11, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
The processing unit 16 executes various functional applications and page display by running programs stored in the system memory 28, for example, implementing the data processing method provided in the present embodiment, including:
responding to the data storage requirement of the first service, and acquiring the service data to be stored in the first format generated by the first service;
converting the business data to be stored from a first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode;
responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage;
and converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service.
Of course, those skilled in the art will appreciate that the processor may also implement the technical solution of the data processing method provided in any embodiment of the present invention.
The present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data processing method such as provided by the present invention, comprising:
responding to the data storage requirement of the first service, and acquiring the service data to be stored in the first format generated by the first service;
Converting the business data to be stored from a first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode;
responding to the data acquisition requirement of the second service, and acquiring intermediate service data from the target storage;
and converting the intermediate service data from the target format to the second format to obtain target service data, and feeding back the target service data to the second service.
The computer storage media of the present embodiments may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
It will be appreciated by those of ordinary skill in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be centralized on a single computing device, or distributed over a network of computing devices, or they may alternatively be implemented in program code executable by a computer device, such that they are stored in a memory device and executed by the computing device, or they may be separately fabricated as individual integrated circuit modules, or multiple modules or steps within them may be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
In addition, the technical scheme of the invention can acquire, store, use, process and the like the data, which accords with the relevant regulations of national laws and regulations.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.

Claims (10)

1. A method of data processing, comprising:
responding to the data storage requirement of a first service, and acquiring service data to be stored in a first format generated by the first service;
converting the business data to be stored from the first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode;
responding to the data acquisition requirement of the second service, and acquiring the intermediate service data from the target storage;
and converting the intermediate service data from the target format to a second format to obtain target service data, and feeding back the target service data to the second service.
2. The data processing method according to claim 1, further comprising, before said converting said traffic data to be stored from said first format to a target format:
determining whether the first format is the same as the target format, and triggering the conversion of the business data to be stored from the first format to the target format when the first format is different from the target format;
before said converting said intermediate traffic data from said target format to a second format, further comprising:
Determining whether the second format is the same as the target format, and triggering execution of the conversion of the intermediate service data from the target format to the second format when the second format is different from the target format.
3. The data processing method according to claim 1, wherein storing the intermediate service data in the target storage according to the structured storage manner includes:
calling a preset storage interface of the target storage to store the intermediate service data into the target storage according to a structured storage mode;
the obtaining the intermediate service data from the target storage includes: and calling a preset access interface of the target storage to acquire the intermediate service data from the target storage.
4. The data processing method of claim 1, wherein the method further comprises:
and deleting the intermediate service data stored in the target storage.
5. The data processing method according to claim 4, further comprising, before said deleting said intermediate service data stored in said target storage:
determining whether a preset deletion condition is reached;
And triggering and executing the deletion of the intermediate service data stored in the target storage when the preset deletion condition is reached.
6. The data processing method according to claim 1, further comprising, after said storing said intermediate service data in a structured storage manner in a target storage:
and setting access rights for the intermediate service data according to the indication of the first service.
7. The data processing method of claim 6, further comprising, prior to said retrieving said intermediate service data from said target store:
determining whether the second service has access rights to the intermediate service data;
and triggering the execution of the acquisition of the intermediate service data from the target storage when the second service has the access right of the intermediate service data.
8. A data processing apparatus, comprising:
the first acquisition module is used for responding to the data storage requirement of a first service and acquiring service data to be stored in a first format generated by the first service;
the storage module is used for converting the business data to be stored from the first format into a target format to obtain intermediate business data, and storing the intermediate business data into a target storage according to a structured storage mode;
The second acquisition module is used for responding to the data acquisition requirement of the second service and acquiring the intermediate service data from the target storage;
and the feedback module is used for converting the intermediate service data from the target format to a second format to obtain target service data and feeding back the target service data to the second service.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data processing method according to any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the data processing method according to any one of claims 1 to 7.
CN202311639285.9A 2023-12-01 2023-12-01 Data processing method, device, equipment and storage medium Pending CN117609352A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311639285.9A CN117609352A (en) 2023-12-01 2023-12-01 Data processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311639285.9A CN117609352A (en) 2023-12-01 2023-12-01 Data processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117609352A true CN117609352A (en) 2024-02-27

Family

ID=89953190

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311639285.9A Pending CN117609352A (en) 2023-12-01 2023-12-01 Data processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117609352A (en)

Similar Documents

Publication Publication Date Title
CN108804140B (en) Batch instruction analysis method, device and equipment
CN111400061A (en) Data processing method and system
US11314451B2 (en) Method and apparatus for storing data
CN110688096B (en) Method and device for constructing application program containing plug-in, medium and electronic equipment
CN116719622A (en) Service flow arranging method and service flow arranging system
CN112288577A (en) Transaction processing method and device for distributed service, electronic equipment and medium
CN111966653A (en) Data processing method, device, server and storage medium for micro-service call link
CN115357761A (en) Link tracking method and device, electronic equipment and storage medium
CN111241180A (en) Information processing method, device, equipment and storage medium
CN112948306A (en) Method and device for expanding system interface
CN113641633B (en) File processing method, apparatus, electronic device, medium, and computer program
CN117609352A (en) Data processing method, device, equipment and storage medium
CN114780361A (en) Log generation method, device, computer system and readable storage medium
CN115460265A (en) Interface calling method, device, equipment and medium
CN117395314A (en) Request processing method, request processing device, electronic equipment and storage medium
US11995202B2 (en) Computer system and data access control method
CN116757689A (en) Request processing method and device, electronic equipment and computer readable storage medium
CN116071075A (en) Transaction integrity verification method, apparatus, device, medium and program product
CN114895980A (en) Application program running method and device, electronic equipment and storage medium
CN116975200A (en) Method, device, equipment and medium for controlling working state of server
CN114818918A (en) Service processing method and device, electronic equipment and storage medium
CN116881736A (en) Information matching method, device, equipment and storage medium
CN116089411A (en) Data checking method, device, equipment and storage medium
CN116257375A (en) Kafka data automatic stream processing method and device
CN116821251A (en) Data processing method, data processing device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination