CN116522200A - Data circulation method, device, equipment, storage medium and product - Google Patents
Data circulation method, device, equipment, storage medium and product Download PDFInfo
- Publication number
- CN116522200A CN116522200A CN202310457368.XA CN202310457368A CN116522200A CN 116522200 A CN116522200 A CN 116522200A CN 202310457368 A CN202310457368 A CN 202310457368A CN 116522200 A CN116522200 A CN 116522200A
- Authority
- CN
- China
- Prior art keywords
- file
- electronic data
- data file
- circulated
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000003860 storage Methods 0.000 title claims abstract description 43
- 238000002360 preparation method Methods 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 21
- 238000001514 detection method Methods 0.000 claims description 10
- 230000000977 initiatory effect Effects 0.000 claims description 5
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 7
- 238000007726 management method Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 230000001960 triggered effect Effects 0.000 description 7
- 230000006870 function Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000012550 audit Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Human Computer Interaction (AREA)
- Life Sciences & Earth Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data transfer method, a device, equipment, a storage medium and a product. The invention relates to the technical field of big data. The method comprises the following steps: acquiring an electronic data file to be circulated; determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated; and based on the data processing logic, the electronic data file to be circulated is circulated to obtain a target electronic data file. According to the technical scheme, the automatic streaming of the electronic data file is realized, and the technical effect of improving the data processing efficiency can be achieved.
Description
Technical Field
The embodiment of the invention relates to the technical field of big data, in particular to a data circulation method, a device, equipment, a storage medium and a product.
Background
For today, where the internet is highly developed, tens of thousands of electronic data are produced every day.
At present, in the electronic data processing process, related technicians are required to collect electronic data files and manually perform various processing operations on different electronic data files, so that the problem of low data processing efficiency exists.
Disclosure of Invention
The embodiment of the invention provides a data circulation method, a device, equipment, a storage medium and a product, which are used for solving the problem of low data processing efficiency.
In a first aspect, an embodiment of the present invention provides a data circulation method, including:
acquiring an electronic data file to be circulated;
determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated;
and based on the data processing logic, the electronic data file to be circulated is circulated to obtain a target electronic data file.
In a second aspect, an embodiment of the present invention further provides a data circulation device, where the device includes:
the electronic data file acquisition module is used for acquiring electronic data files to be transferred;
the data processing logic determining module is used for determining the file type of the electronic data file to be circulated and determining data processing logic according to the file type of the electronic data file to be circulated;
and the electronic data file circulation module is used for circulating the electronic data file to be circulated based on the data processing logic to obtain a target electronic data file.
In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the processor implements a data circulation method according to any one of the embodiments of the present invention when the processor executes the program.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements a data transfer method according to any one of the embodiments of the present invention.
In a fifth aspect, embodiments of the present invention further provide a computer program product comprising a computer program which, when executed by a processor, implements a data transfer method according to any of the embodiments of the present invention.
In the embodiment of the invention, the file type of the electronic data file to be circulated is determined by acquiring the electronic data file to be circulated, the data processing logic is determined according to the file type of the electronic data file to be circulated, and the electronic data file to be circulated is circulated based on the data processing logic, so that the target electronic data file is obtained. According to the technical scheme, the automatic circulation of the electronic data file is realized, and the technical effect of improving the data processing efficiency can be achieved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a data flow method according to an embodiment of the present invention;
fig. 2 is a flowchart of a data flow method according to an embodiment of the present invention;
fig. 3 is a flowchart of a data flow method according to an embodiment of the present invention;
fig. 4 is a flowchart of a data flow method according to an embodiment of the present invention;
fig. 5 is a flowchart of a data circulation method according to an embodiment of the present invention;
FIG. 6 is a flowchart of a method for transferring backup files of a database according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a data circulation device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present invention, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance. The data acquisition, storage, use, processing and the like in the technical scheme meet the relevant regulations of national laws and regulations.
Fig. 1 is a flowchart of a data transfer method according to an embodiment of the present invention, where the method may be applied to an automatic electronic data file transfer situation, and the method may be performed by a data transfer device, where the data transfer device may be implemented in a hardware and/or software form, and the data transfer device may be configured in a computer terminal, a server, or other devices. As shown in fig. 1, the method includes:
S110, acquiring an electronic data file to be circulated.
In this embodiment, the electronic data file refers to electronic data to be circulated, which may be data of industries such as finance, internet, manufacturing industry, and the like.
For example, taking an audit scenario as an example, the electronic data file may be unit financial data of an audited unit, specifically, the unit financial data may be obtained from a preset storage path of the electronic device, or may be obtained from a device or cloud connected to the electronic device in a communication manner, which is not limited herein.
S120, determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated.
In this embodiment, the file type of the electronic data file may include, but is not limited to, a text file type, a data file type, a streaming media file type, or the like. It may be appreciated that there is a difference in the data processing procedure of the electronic data files of different file types, and the embodiment may invoke corresponding data processing logic according to the file types of the electronic data files to be circulated, where the data processing logic may include one or more events with a preset execution sequence, and the events may include, but are not limited to, a restore event, a resource preparation event, a file circulation event, an end event, and so on.
Specifically, the file type of the electronic data file to be circulated can be determined according to the file suffix name, the file content and the like of the electronic data file, so that the data processing logic corresponding to the file type of the electronic data file to be circulated can be called, the automatic calling of the data processing logic is realized, and the data processing efficiency is improved.
And S130, based on the data processing logic, the electronic data file to be circulated is circulated, and a target electronic data file is obtained.
In this embodiment, each event in the data processing logic may be sequentially executed according to a preset execution sequence, so as to implement the circulation of the electronic data file to be circulated, and obtain the target electronic data file. The target electronic data file is an electronic data file after the data processing logic.
In the embodiment of the invention, the file type of the electronic data file to be circulated is determined by acquiring the electronic data file to be circulated, the data processing logic is determined according to the file type of the electronic data file to be circulated, and the electronic data file to be circulated is circulated based on the data processing logic, so that the target electronic data file is obtained. According to the technical scheme, the automatic circulation of the electronic data file is realized, and the technical effect of improving the data processing efficiency can be achieved.
Fig. 2 is a flowchart of a data transfer method according to an embodiment of the present invention, where the method according to the present embodiment may be combined with each of the alternatives in the data transfer method provided in the foregoing embodiment. The data streaming method provided by the embodiment is further optimized. Optionally, the obtaining the electronic data file to be circulated includes: and receiving the electronic data file to be circulated, which is uploaded by the user.
As shown in fig. 2, the method includes:
s210, receiving an electronic data file to be circulated, which is uploaded by a user.
S220, determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated.
And S230, based on the data processing logic, the electronic data file to be circulated is circulated, and a target electronic data file is obtained.
In this embodiment, the electronic device may receive the electronic data file to be circulated uploaded by the user, so that the electronic device determines a file type of the electronic data file to be circulated uploaded by the user, further determines a data processing logic according to the file type of the electronic data file to be circulated, and further performs circulation on the electronic data file to be circulated based on the data processing logic, thereby obtaining a target electronic data file, and implementing automatic circulation of the electronic data file.
It should be noted that the data circulation method may be implemented by a plurality of components, where the components may include a driving component, a storage component, a circulation component, and a resource management component. The driving component is an event driver and is used for driving an event, the storage component is used for providing interface information, the circulation component is used for executing a data processing process, and the resource management component is used for acquiring resource information. The user can combine, disassemble and expand each component according to the data flow requirement, so that the data automation flow of the personalized data flow requirement is realized.
Specifically, the electronic device may initiate a file upload event through the electronic data stream interface; under the condition that the driving component receives a file uploading event, acquiring storage resource information through the resource management component; and calling an uploading interface of the storage component based on the storage resource information, and uploading the electronic data file to be circulated through the uploading interface of the storage component.
The storage resource information may include IP, port information, and the like, among others.
By taking Oracle database backup file flow as an example, a user can initiate a file uploading event on an Oracle database backup file through an electronic data flow interface, and after receiving the file uploading event, the driving component calls the resource management component to acquire storage resource information such as IP (Internet protocol) and port information, and further can call an uploading interface of the storage component according to the storage resource information so as to upload the file to be uploaded by the Oracle database backup file.
In this embodiment, the electronic device may receive the electronic data file to be circulated uploaded by the user, so that the electronic device determines a file type of the electronic data file to be circulated uploaded by the user, further determines a data processing logic according to the file type of the electronic data file to be circulated, and further performs circulation on the electronic data file to be circulated based on the data processing logic, thereby obtaining a target electronic data file, and realizing automatic circulation of the electronic data file.
Fig. 3 is a flowchart of a data transfer method according to an embodiment of the present invention, where the method according to the present embodiment may be combined with each of the alternatives in the data transfer method provided in the foregoing embodiment. The data streaming method provided by the embodiment is further optimized. Optionally, the determining the file type of the electronic data file to be circulated includes: and identifying the electronic data file to be transferred based on a preset file type detection method to obtain the file type of the electronic data file to be transferred.
As shown in fig. 3, the method includes:
s310, acquiring an electronic data file to be circulated.
S320, identifying the electronic data file to be circulated based on a preset file type detection method to obtain the file type of the electronic data file to be circulated.
S330, determining data processing logic according to the file type of the electronic data file to be circulated.
S340, based on the data processing logic, the electronic data file to be circulated is circulated, and a target electronic data file is obtained.
In this embodiment, the preset file type detection method may include one or more of the following. Specifically, the type of the electronic data file to be circulated can be obtained by performing type identification on the circulated electronic data file by Tika; the file type of the electronic data file to be circulated can also be determined through the file header information of the electronic data file to be circulated, which is not limited herein.
On the basis of the above embodiments, optionally, identifying the electronic data file to be circulated based on a preset file type detection method to obtain a file type of the electronic data file to be circulated, including: and identifying the electronic data file to be circulated through Tika to obtain the file type of the electronic data file to be circulated.
For the common electronic file types such as TXT, XML, PDF and the like, the electronic data file to be circulated can be identified through Tika, and the file type of the electronic data file to be circulated is obtained.
On the basis of the above embodiments, optionally, identifying the electronic data file to be circulated based on a preset file type detection method to obtain a file type of the electronic data file to be circulated, including: acquiring file header information of an electronic data file to be circulated; and matching the file header information of the electronic data file to be transferred in a rule base to obtain the file type of the electronic data file to be transferred.
For example, for an electronic data file of specific software, the header information of the electronic data file may be extracted and converted into 16-system, and the header information converted into 16-system is matched in a rule base, so as to obtain a file type corresponding to the header information, for example, the header information of an XLS file type is D0CF11E0, the header information of an XML file type is 3C3F786D6C, and the rule base includes a correspondence between the file type and the header information.
In the embodiment of the invention, the electronic data files to be circulated are identified based on the preset file type detection method, so that the file types of the electronic data files to be circulated are obtained, the automatic determination of the file types of the electronic data files is realized, and the data processing efficiency is improved.
Fig. 4 is a flowchart of a data transfer method according to an embodiment of the present invention, where the method according to the embodiment may be combined with each of the alternatives in the data transfer method provided in the foregoing embodiment. The data streaming method provided by the embodiment is further optimized. Optionally, the determining the data processing logic according to the file type of the electronic data file to be circulated includes: triggering a resource preparation event through a driving component under the condition that the file type of the electronic data file to be circulated is determined to be completed; and under the condition that the preparation of the resource is completed, invoking data processing logic according to the file type of the electronic data file to be circulated.
As shown in fig. 4, the method includes:
s410, acquiring an electronic data file to be circulated.
S420, determining the file type of the electronic data file to be circulated.
S430, triggering a resource preparation event through a driving component under the condition that the file type of the electronic data file to be circulated is determined to be completed.
S440, under the condition that resource preparation is completed, invoking data processing logic according to the file type of the electronic data file to be circulated.
S450, based on the data processing logic, the electronic data file to be circulated is circulated, and a target electronic data file is obtained.
In this embodiment, when the determination of the file type of the electronic data file to be circulated is completed, the resource preparation event may be triggered by the driving component, and the resource management component may perform resource preparation corresponding to the file type according to the files of different types. And further, under the condition that the resource preparation is completed, invoking the data processing logic according to the file type of the electronic data file to be circulated.
For example, after the resource management component obtains the file type as the Oracle backup file, oracle version information is judged according to the extracted file header information, and then Oracle available resources are obtained according to the Oracle version information, so as to complete resource preparation and provide a resource foundation for the subsequent data processing process.
On the basis of the above embodiments, optionally, the case where resource preparation is completed includes: transmitting the electronic data file to be circulated to a target server; or, initiating a restoring task of the electronic data file to be circulated.
In an exemplary embodiment, when the electronic data file to be streamed is transmitted to the target server, or the resource preparation such as the restoration task of the electronic data file to be streamed is initiated, the data processing logic is invoked according to the file type of the electronic data file to be streamed, and the data processing step for the electronic data file is initiated.
In the embodiment of the invention, under the condition that the file type of the electronic data file to be circulated is determined to be completed, the driving component triggers the resource preparation event, and further under the condition that the resource preparation is completed, the data processing logic is called according to the file type of the electronic data file to be circulated, thereby realizing the automatic determination of the data processing logic and improving the data processing efficiency.
Fig. 5 is a flowchart of a data transfer method according to an embodiment of the present invention, where the method according to the present embodiment may be combined with each of the alternatives in the data transfer method provided in the foregoing embodiment. The data streaming method provided by the embodiment is further optimized. Optionally, the data processing logic includes a file transfer event, a restore event, and a data processing event; correspondingly, the transferring the electronic data file to be transferred based on the data processing logic to obtain a target electronic data file comprises the following steps: triggering a file transfer event under the condition that resource preparation is completed, and transmitting the electronic data file to be transferred to a server local disk; triggering a restoring event and executing file restoring operation; under the condition that data reduction is completed, feeding back a successful reduction result to the driving assembly; triggering a data processing event under the condition that the driving component receives a successful restoration result; and extracting, converting and loading the electronic data file to be transferred through the transfer component to obtain a target electronic data file.
As shown in fig. 5, the method includes:
s510, acquiring an electronic data file to be circulated.
S520, determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated, wherein the data processing logic comprises a file circulation event, a restoration event and a data processing event.
And S530, triggering a file transfer event under the condition that the resource preparation is completed, and transmitting the electronic data file to be transferred to a server local disk.
S540, triggering a restoring event, executing file restoring operation, and feeding back a restoring success result to the driving assembly under the condition that data restoring is completed.
And S550, triggering a data processing event under the condition that the driving component receives a successful restoration result, and extracting, converting and loading the electronic data file to be circulated through the circulation component to obtain a target electronic data file.
Fig. 6 is a flowchart of a method for transferring backup files of a database according to an embodiment of the present invention. Under the condition that the preparation of the Oracle available resources is completed, a file transfer event is triggered, an electronic data file to be transferred is transmitted to an Oracle service local disk, then a restoration event is triggered, a transfer component executes file restoration operation, a restoration result is fed back to a driving component under the condition that the data restoration is completed, under the condition that the driving component receives restoration success result information, a data processing event is triggered, namely an ETL (Extract-Transform-Load) process is executed, and the electronic data file to be transferred is extracted, converted and loaded through the transfer component, so that a target electronic data file is obtained.
Based on the above embodiments, optionally, the data processing logic further includes an end event; accordingly, after obtaining the target electronic data file, the method further comprises: the driving component triggers an ending event, the circulation component executes the ending event, the driving component judges a data processing result, if the data processing result is failure, the driving component determines a failure reason according to the error log information, and determines a data processing mode based on the failure reason.
After the execution of the ETL process is completed, the driving component triggers an ending event, and then the circulation component executes the ending event, and then the driving component judges a data processing result, and if the data processing result is successful, the circulation is completed; if the data processing result is failure, the driving component can determine the failure reason according to the error log information or the event state, and then select a corresponding data processing mode according to the failure reason so as to complete data circulation.
Based on the foregoing embodiments, optionally, determining a data processing manner based on the failure cause includes: and if the failure cause is a first error cause, carrying out data processing on the electronic data file to be circulated again, wherein the first error cause comprises one or more of mismatch of database versions and mismatch of data formats.
Based on the foregoing embodiments, optionally, determining a data processing manner based on the failure cause includes: if the failure cause is a second error cause, terminating the data processing of the electronic data file to be circulated, wherein the second error cause comprises one or more of file damage, server crash and program unresponsiveness.
For example, for general errors such as mismatching of database versions and inconsistent data formats, data processing of the electronic data file to be circulated can be initiated again until the electronic data file is successfully processed. For fatal errors such as file damage, server crash, no response of programs and the like, the data processing of the electronic data file to be circulated can be stopped.
In the embodiment of the invention, under the condition that the resource preparation is completed, a file transfer event is triggered, an electronic data file to be transferred is transmitted to a local disk of a server, a restore event is further triggered, file restore operation is executed, and a restore success result is further fed back to a driving component under the condition that the data restore is completed; under the condition that the drive assembly receives a successful restoration result, a data processing event is triggered, and then the electronic data file to be circulated is extracted, converted and loaded through the circulation assembly, so that a target electronic data file is obtained, automatic circulation of data is realized, and data processing efficiency is improved.
Fig. 7 is a schematic structural diagram of a data circulation device according to an embodiment of the present invention. As shown in fig. 7, the apparatus includes:
an electronic data file obtaining module 610, configured to obtain an electronic data file to be transferred;
a data processing logic determining module 620, configured to determine a file type of the electronic data file to be streamed, and determine data processing logic according to the file type of the electronic data file to be streamed;
and the electronic data file circulation module 630 is configured to circulate the electronic data file to be circulated based on the data processing logic, so as to obtain a target electronic data file.
In some alternative embodiments, the electronic data file acquisition module 610 includes:
and the electronic data file receiving unit is used for receiving the electronic data file to be circulated, which is uploaded by the user.
In some alternative embodiments, the electronic data file receiving unit is further configured to:
initiating a file uploading event through an electronic data stream interface;
under the condition that the driving component receives a file uploading event, acquiring storage resource information through the resource management component;
and calling an uploading interface of the storage component based on the storage resource information, and uploading the electronic data file to be circulated through the uploading interface of the storage component.
In some alternative embodiments, the data processing logic determination module 620 includes:
the file type identification unit is used for identifying the electronic data file to be circulated based on a preset file type detection method to obtain the file type of the electronic data file to be circulated.
In some alternative embodiments, the file type identification unit is further configured to:
and identifying the electronic data file to be circulated through Tika to obtain the file type of the electronic data file to be circulated.
In some alternative embodiments, the file type identification unit is further configured to:
acquiring file header information of the electronic data file to be circulated;
and matching the file header information of the electronic data file to be circulated in a rule base to obtain the file type of the electronic data file to be circulated.
In some alternative embodiments, the data processing logic determination module 620 is further configured to:
triggering a resource preparation event through a driving component under the condition that the file type of the electronic data file to be circulated is determined to be completed;
and under the condition that the resource preparation is completed, invoking data processing logic according to the file type of the electronic data file to be circulated.
In some alternative embodiments, the resource preparation completion comprises:
transmitting the electronic data file to be circulated to a target server;
or, initiating a restoring task of the electronic data file to be circulated.
In some alternative embodiments, the data processing logic includes file transfer events, restore events, and data processing events; the electronic data file circulation module 630 includes:
the electronic data file storage unit is used for triggering a file transfer event and transmitting the electronic data file to be transferred to a server local disk under the condition that the resource preparation is completed;
the electronic data file restoring unit is used for triggering a restoring event and executing file restoring operation, and feeding back a restoring success result to the driving assembly under the condition that data restoring is completed;
and the electronic data file processing unit is used for triggering a data processing event under the condition that the driving assembly receives a successful restoration result, and extracting, converting and loading the electronic data file to be circulated through the circulation assembly to obtain a target electronic data file.
In some alternative embodiments, the data processing logic further comprises an end event; the data circulation device further comprises:
The circulation ending module is used for triggering an ending event by the driving module, executing the ending event by the circulation module, judging a data processing result by the driving module, determining a failure reason by the driving module according to the error log information if the data processing result is failure, and determining a data processing mode based on the failure reason.
In some alternative embodiments, the end of circulation module is further configured to:
and if the failure cause is a first error cause, carrying out data processing on the electronic data file to be circulated again, wherein the first error cause comprises one or more of mismatch of database versions and mismatch of data formats.
In some alternative embodiments, the end of circulation module is further configured to:
and if the failure cause is a second error cause, stopping data processing on the electronic data file to be circulated, wherein the second error cause comprises one or more of file damage, server crash and program unresponsiveness.
The data transfer device provided by the embodiment of the invention can execute the data transfer method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure. Referring now to fig. 8, a schematic diagram of an electronic device (e.g., a terminal device or server in fig. 8) 500 suitable for use in implementing embodiments of the present disclosure is shown. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and the like, and stationary terminals such as digital TVs, desktop computers, and the like. The electronic device shown in fig. 8 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 8, the electronic device 500 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 501, which may perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit/output (I/O) interface 505 is also connected to bus 504.
In general, the following devices may be connected to the I/O interface 505: input devices 506 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 507 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 508 including, for example, magnetic tape, hard disk, etc.; and communication means 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 8 shows an electronic device 500 having various means, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 509, or from the storage means 508, or from the ROM 502. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by the processing device 501.
The names of messages or information interacted between the various devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.
The electronic device provided by the embodiment of the present disclosure and the data transfer method provided by the foregoing embodiment belong to the same inventive concept, and technical details not described in detail in the present embodiment may be referred to the foregoing embodiment, and the present embodiment has the same beneficial effects as the foregoing embodiment.
The present disclosure provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the data streaming method provided by the above embodiments.
It should be noted that the computer readable medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (HyperText TransferProtocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to:
acquiring an electronic data file to be circulated;
determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated;
and based on the data processing logic, the electronic data file to be circulated is circulated to obtain a target electronic data file.
Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
Embodiments of the present invention also provide a computer program product comprising a computer program which, when executed by a processor, implements a data flow method as provided by any of the embodiments of the present application.
Computer program product in the implementation, the computer program code for carrying out operations of the present invention may be written in one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider).
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (16)
1. A method of data streaming, comprising:
acquiring an electronic data file to be circulated;
determining the file type of the electronic data file to be circulated, and determining data processing logic according to the file type of the electronic data file to be circulated;
and based on the data processing logic, the electronic data file to be circulated is circulated to obtain a target electronic data file.
2. The method of claim 1, wherein the obtaining the electronic data file to be streamed comprises:
And receiving the electronic data file to be circulated, which is uploaded by the user.
3. The method of claim 2, wherein receiving the electronic data file to be streamed uploaded by the user comprises:
initiating a file uploading event through an electronic data stream interface;
under the condition that the driving component receives a file uploading event, acquiring storage resource information through the resource management component;
and calling an uploading interface of the storage component based on the storage resource information, and uploading the electronic data file to be circulated through the uploading interface of the storage component.
4. The method of claim 1, wherein the determining the file type of the electronic data file to be streamed comprises:
and identifying the electronic data file to be transferred based on a preset file type detection method to obtain the file type of the electronic data file to be transferred.
5. The method according to claim 4, wherein the identifying the electronic data file to be transferred based on the preset file type detection method to obtain a file type of the electronic data file to be transferred includes:
and identifying the electronic data file to be circulated through Tika to obtain the file type of the electronic data file to be circulated.
6. The method according to claim 4, wherein the identifying the electronic data file to be transferred based on the preset file type detection method to obtain a file type of the electronic data file to be transferred includes:
acquiring file header information of the electronic data file to be circulated;
and matching the file header information of the electronic data file to be circulated in a rule base to obtain the file type of the electronic data file to be circulated.
7. The method of claim 1, wherein said determining data processing logic based on a file type of said electronic data file to be streamed comprises:
triggering a resource preparation event through a driving component under the condition that the file type of the electronic data file to be circulated is determined to be completed;
and under the condition that the resource preparation is completed, invoking data processing logic according to the file type of the electronic data file to be circulated.
8. The method of claim 7, wherein the condition of resource preparation completion comprises:
transmitting the electronic data file to be circulated to a target server;
or, initiating a restoring task of the electronic data file to be circulated.
9. The method of claim 1, wherein the data processing logic comprises a file stream event, a restore event, and a data processing event;
correspondingly, the transferring the electronic data file to be transferred based on the data processing logic to obtain a target electronic data file comprises the following steps:
triggering a file transfer event under the condition that resource preparation is completed, and transmitting the electronic data file to be transferred to a server local disk;
triggering a restoring event, executing file restoring operation, and feeding back a restoring success result to the driving assembly under the condition that data restoring is completed;
and under the condition that the driving component receives a successful restoration result, triggering a data processing event, and extracting, converting and loading the electronic data file to be circulated through the circulation component to obtain a target electronic data file.
10. The method of claim 9, wherein the data processing logic further comprises an end event;
accordingly, after obtaining the target electronic data file, the method further comprises:
the driving component triggers an ending event, the circulation component executes the ending event, the driving component judges a data processing result, if the data processing result is failure, the driving component determines a failure reason according to the error log information, and determines a data processing mode based on the failure reason.
11. The method of claim 10, wherein the determining a data processing manner based on the failure cause comprises:
and if the failure cause is a first error cause, carrying out data processing on the electronic data file to be circulated again, wherein the first error cause comprises one or more of mismatch of database versions and mismatch of data formats.
12. The method of claim 10, wherein the determining a data processing manner based on the failure cause comprises:
and if the failure cause is a second error cause, stopping data processing on the electronic data file to be circulated, wherein the second error cause comprises one or more of file damage, server crash and program unresponsiveness.
13. A data streaming apparatus, comprising:
the electronic data file acquisition module is used for acquiring electronic data files to be transferred;
the data processing logic determining module is used for determining the file type of the electronic data file to be circulated and determining data processing logic according to the file type of the electronic data file to be circulated;
and the electronic data file circulation module is used for circulating the electronic data file to be circulated based on the data processing logic to obtain a target electronic data file.
14. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor implements the data transfer method of any of claims 1-12 when executing the computer program.
15. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements a data flow method as claimed in any one of claims 1-12.
16. A computer program product comprising a computer program which, when executed by a processor, implements the data flow method according to any of claims 1-12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310457368.XA CN116522200A (en) | 2023-04-25 | 2023-04-25 | Data circulation method, device, equipment, storage medium and product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310457368.XA CN116522200A (en) | 2023-04-25 | 2023-04-25 | Data circulation method, device, equipment, storage medium and product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116522200A true CN116522200A (en) | 2023-08-01 |
Family
ID=87398780
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310457368.XA Pending CN116522200A (en) | 2023-04-25 | 2023-04-25 | Data circulation method, device, equipment, storage medium and product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116522200A (en) |
-
2023
- 2023-04-25 CN CN202310457368.XA patent/CN116522200A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11269705B2 (en) | Method and apparatus for outputting information | |
| CN107644075B (en) | Method and device for collecting page information | |
| CN112596720A (en) | Service operation method and device, electronic equipment and computer storage medium | |
| CN110727566A (en) | Method, device, medium and electronic equipment for acquiring process crash information | |
| CN110908921A (en) | Game problem positioning method and device, electronic equipment and storage medium | |
| CN113807056B (en) | Document name sequence error correction method, device and equipment | |
| CN115600964A (en) | Voice approval method and device and related equipment | |
| CN112422648B (en) | Data synchronization method and system | |
| CN116522200A (en) | Data circulation method, device, equipment, storage medium and product | |
| CN111460020B (en) | Method, device, electronic equipment and medium for resolving message | |
| CN111290873B (en) | Fault processing method and device | |
| CN112783903B (en) | Method and device for generating update log | |
| CN111367590A (en) | Interrupt event processing method and device | |
| CN113704079A (en) | Interface testing method and device based on Protobuf | |
| US20240152504A1 (en) | Data interaction method, apparatus, and electronic device | |
| CN114356788B (en) | Application program detection method, device, equipment and medium based on user information | |
| CN112084440B (en) | Data verification method, device, electronic equipment and computer readable medium | |
| CN110099122B (en) | Method and apparatus for sending network request | |
| CN110855767B (en) | Method, device, equipment and storage medium for responding operation request | |
| CN116739809A (en) | Financial statement processing method, device, equipment, storage medium and product | |
| CN116361032A (en) | Information processing method and device and electronic equipment | |
| CN114039970A (en) | Information downloading method and device, electronic equipment and computer readable medium | |
| CN117493091A (en) | Storage system fault injection method, device, equipment and storage medium | |
| CN116775001A (en) | Method, device, electronic equipment and storage medium for processing application program | |
| CN114090139A (en) | Page processing method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |