CN110865990B - Non-real-time data exchange method, system and computer equipment - Google Patents
Non-real-time data exchange method, system and computer equipment Download PDFInfo
- Publication number
- CN110865990B CN110865990B CN201910968954.4A CN201910968954A CN110865990B CN 110865990 B CN110865990 B CN 110865990B CN 201910968954 A CN201910968954 A CN 201910968954A CN 110865990 B CN110865990 B CN 110865990B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- real
- batch
- receiving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 238000012545 processing Methods 0.000 claims abstract description 64
- 238000012986 modification Methods 0.000 claims description 30
- 230000004048 modification Effects 0.000 claims description 30
- 230000008569 process Effects 0.000 claims description 20
- 239000000725 suspension Substances 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 9
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000003672 processing method Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 abstract description 2
- 238000010926 purge Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 7
- 238000012795 verification Methods 0.000 description 7
- 238000007781 pre-processing Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 229920006395 saturated elastomer Polymers 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/08—Insurance
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a non-real-time data exchange method, which comprises the following steps: acquiring initial data from source equipment, wherein the initial data are pre-marked and classified data; loading the initial data into a receiving directory; performing data processing on the initial data in the receiving catalog to obtain target data, and loading the target data into the release catalog; and publishing the target data to the target device through a publishing directory. According to the embodiment of the invention, the data supplier and the data acquirer are decoupled through non-real-time data exchange, so that efficient data interaction is realized, the data exchange efficiency is further improved, and the consumption and trouble caused by data exchange are reduced.
Description
Technical Field
Embodiments of the present invention relate to the field of data processing, and in particular, to a non-real-time data exchange method, a system, a computer device, and a computer readable storage medium.
Background
At present, a plurality of systems in an enterprise system mutually call a borrowing port to transfer data, but the service load is continuously increased along with time, and the original architecture cannot bear the current service load due to the original design. Some server clusters already bear a large amount of service calculation tasks, the interface throughput is quite busy and green, and other clusters still continuously request the same data, so that the interface IO and system resources are saturated; meanwhile, the clusters repeatedly call interface request data and mutually wait for data of the other party; the prior system storage modes are not uniform, a direct-connected storage DAS and a storage area network SAN coexist, each system has independent data storage physical addresses, so that the data management and maintenance are inconvenient, each time of data migration and updating are manually processed, a supplier and a acquirer are required to be tightly coupled with each other through a platform to realize functions, and huge consumption is very troublesome.
Therefore, how to decouple the provider and the acquirer and realize efficient data exchange, thereby further improving the data exchange efficiency and reducing the computer resource consumption caused by data exchange becomes one of the technical problems to be solved at present.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a non-real-time data exchange method, system, computer device and computer readable storage medium, so as to solve the technical problems that the data provider and the data demander are tightly coupled to each other through a platform to realize the data exchange, and the data exchange consumes huge and very troublesome.
To achieve the above object, an embodiment of the present invention provides a non-real-time data exchange method, including:
acquiring initial data from source equipment, wherein the initial data are pre-marked and classified data;
Loading the initial data into a receiving directory;
Performing data processing on the initial data in the receiving catalog to obtain target data, and loading the target data into the release catalog; and
And publishing the target data to the target device through a publishing directory.
Illustratively, the initial data includes full-volume data and/or batch data, and the receiving catalog includes a near real-time receiving catalog and a batch receiving catalog; loading the initial data into a receiving directory, comprising:
judging whether a demand instruction of target equipment is received or not, wherein the demand instruction comprises a full-increment demand instruction;
if the demand instruction is received, correspondingly loading the full data into the quasi-real-time receiving catalog, and correspondingly loading the batch data into the batch receiving catalog; and
And if the demand instruction is not received, loading the full data and the batch data into a batch receiving catalog.
Illustratively, the target data includes target full-volume data and/or target batch data, and the issue catalog includes a near real-time issue catalog and an incremental issue catalog; loading the target data into a release catalog, including:
if the demand instruction is received, correspondingly issuing the target full-volume data to the quasi-real-time issuing catalog, and correspondingly issuing the target batch data to the increment issuing catalog;
And if the demand instruction is not received, the target full-volume data and the batch data are issued to an increment issue catalog.
Exemplary, the data processing of the initial data in the receiving directory to obtain target data includes:
monitoring the processed process of the full data or batch data;
When the field information modification event occurs in the processed process, the processing method comprises the following steps: generating alert information and writing the alert information into a modification log component, wherein the modification log component is used for informing the source device that the initial data has been modified, and the alert information comprises detailed information before modification and detailed information after modification;
Wherein the field information modification event includes a data modification event and a purge event including a purge event for a field exceeding a preset length and a purge event for data including a preset character.
Exemplary, the data processing of the initial data in the receiving directory to obtain target data further includes:
when the processing failure event occurs in the processed process is monitored: suspending the data processing operation; generating data processing suspending operation information, writing the data processing suspending operation information into a failure log component, wherein the failure log component is used for informing the source equipment that the initial data uploading fails;
Wherein the data processing operation suspension information comprises reason description information triggering the data processing operation suspension; the processing failure event includes: overload event caused by data preloading exceeding a preset loading range.
Illustratively, publishing the target data to the target device via the publishing directory includes:
when the target data is detected to be loaded into the release catalog for release, generating a notification release completion file and a notification acquisition data file;
The notification release completion file is sent to source equipment and used for notifying the source equipment that release is completed; and
And sending the notification acquisition data file to the target equipment, and notifying the target equipment to acquire the target data at a preset frequency.
To achieve the above object, an embodiment of the present invention further provides a non-real-time data exchange system, including:
The acquisition module is used for acquiring initial data from the source equipment, wherein the initial data are pre-marked and classified data;
the loading module is used for loading the initial data to a receiving catalog;
the processing module is used for carrying out data processing on the initial data in the receiving catalog to obtain target data, and loading the target data into the release catalog; and
And the issuing module is used for issuing the target data to the target equipment through the issuing catalogue.
Illustratively, the initial data includes full-volume data and/or batch data, and the receiving catalog includes a near real-time receiving catalog and a batch receiving catalog; the loading module is further configured to:
judging whether a demand instruction of target equipment is received or not, wherein the demand instruction comprises a full-increment demand instruction;
if the demand instruction is received, correspondingly loading the full data into the quasi-real-time receiving catalog, and correspondingly loading the batch data into the batch receiving catalog; and
And if the demand instruction is not received, loading the full data and the batch data into a batch receiving catalog.
To achieve the above object, an embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the computer program is executed by the processor to implement the steps of the non-real-time data exchange method as described above.
To achieve the above object, an embodiment of the present invention further provides a computer-readable storage medium having stored therein a computer program executable by at least one processor to cause the at least one processor to perform the steps of the non-real-time data exchange method as described above.
The non-real-time data exchange method, the system, the computer equipment and the computer readable storage medium provided by the embodiment of the invention provide an effective exchange method for data; by the non-real-time data exchange method, the provider and the acquirer are decoupled, and efficient data interaction is realized, so that the data exchange efficiency is further improved, and the computer resource consumption caused by the data exchange is reduced.
Drawings
Fig. 1 is an environmental application schematic diagram of a non-real-time data exchange method according to an embodiment of the present invention.
Fig. 2 is a flow chart of a non-real-time data exchange method according to an embodiment of the invention.
Fig. 3 is a schematic diagram of a program module of a non-real-time data exchange system according to a second embodiment of the present invention.
Fig. 4 is a schematic diagram of a hardware structure of a third embodiment of the computer device of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the description of "first", "second", etc. in this disclosure is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.
Fig. 1 schematically shows an environmental application diagram of a non-real-time data exchange according to a first embodiment of the application. In an exemplary embodiment, the computer device 2 may connect to a plurality of source devices 4 and to a plurality of target devices 6 over a network 8.
The computer device 2 may be configured with a receive directory, a release directory, and a data processing layer. The data processing layer can comprise a quasi-real-time exchange module, a batch exchange module, a data merging and de-duplication module, a data test transfer module, a central database and the like.
The receiving catalogue is connected with the issuing catalogue through a data processing layer; the receiving catalogs comprise a quasi-real-time receiving catalogs and a batch receiving catalogs, and the publishing catalogs comprise a quasi-real-time publishing catalogs and an incremental publishing catalogs.
The quasi real-time receiving catalogue is connected with the quasi real-time publishing catalogue through a quasi real-time exchange module;
The batch receiving catalogue is connected with an increment release catalogue through the batch exchange module;
The data merging and de-duplication module is connected with the data test transfer module, and the quasi real-time exchange module is connected with the central database through the data merging and de-duplication module and the data test transfer module;
and the batch exchange module is connected with the central database through the data test transfer module.
The quasi real-time exchange module comprises a file analysis layer and a full-quantity file and processing layer. The batch exchange module comprises: a batch verification layer, a batch file preprocessing layer and a data de-duplication layer.
Source device 4, as a provider system, may be an electronic device such as a server or a server cluster for providing data including vehicle insurance underwriting data, vehicle insurance claim data, personal insurance data, and reconfirm data.
The target device 6, as a demand system, may be an electronic device for receiving target data, such as a server or a server cluster, or the like.
The network 8 may be the Internet (Internet), a wide area network (wide area network, WAN), a local area network, or the like.
In the following embodiments, an exemplary description will be made with the computer device 2 as an execution subject.
Example 1
Referring to fig. 2, a flow chart of the steps of a non-real-time data exchange method according to an embodiment of the present invention is shown. It will be appreciated that the flow charts in the method embodiments are not intended to limit the order in which the steps are performed. An exemplary description will be made below with the computer device 2 as an execution subject. Specifically, the following is described.
In step S100, initial data, which is data that is labeled and classified in advance, is acquired from the source device 4.
By way of example, the initial data may include vehicle insurance underwriting data, vehicle insurance claim data, personal insurance data, and reinsurance data.
Illustratively, the initial data includes a signal file (.ok or.ok.ing), a system number, a batch file list. The initial data of the same batch is the front-back data dependency relationship; the initial data is divided into batch data when the initial data has a dependency relationship, the batch data can be serially executed through a receiving-processing-trial loading-publishing sequence, and processes of the serial execution can be parallel; if the initial data has no dependency relationship, the initial data is divided into full data, processes can be executed in parallel, the maximum process number can be set in configuration, and the concurrency can be controlled.
Step S102, the initial data is loaded to a receiving catalog.
In an exemplary embodiment, the initial data includes full data or/and batch data, and the receive catalog includes a near real-time receive catalog and a batch receive catalog.
The step S102 may further include:
Step S102A, judging whether a demand instruction of the target equipment 6 is received, wherein the demand instruction comprises a full-increment demand instruction;
Step S102B, if the demand instruction is received, the full-volume data is correspondingly loaded into the quasi-real-time receiving catalog, and the batch data is correspondingly loaded into the batch receiving catalog; and
Step S102C, if the demand instruction is not received, the full data and the batch data are loaded into a batch receiving catalog.
Illustratively, for the target device 6, data loading does not distinguish between bulk and full, both are deblocked in bulk; and if yes, the method is changed, and if not, the method is inserted. If the target device 6 has a full increment requirement, the full increment logic needs to be implemented in the increment data file of the source device 4; the data is generally provided in a full-scale manner as a basic configuration table with very small data volume and little variation.
Step S104, carrying out data processing on the initial data in the receiving catalog to obtain target data, and loading the target data into the release catalog.
In an exemplary embodiment, the target data includes target full volume data or/and target batch data, and the issue catalog includes a near real-time issue catalog and an incremental issue catalog.
The step S104 may further include:
Step S104A1, if the demand instruction is received, correspondingly issuing the target full-volume data to the quasi-real-time issuing catalog and correspondingly issuing the target batch data to the increment issuing catalog;
Step S104A2, if the demand instruction is not received, the target full-volume data and the batch data are issued to an increment issue catalog.
Illustratively, the full data and the batch data are subjected to different data processing.
The full-volume data is processed by a near-real-time processing module, the full-volume data (i.e., the near-real-time data) is analyzed by the file analysis layer, and the analyzed full-volume data is subjected to the file preprocessing layer to obtain target full-volume data.
The batch data is processed through a batch data processing module, and the batch data sequentially enters a batch verification layer, a batch file preprocessing layer and a data deduplication layer to perform batch verification operation, file preprocessing operation and data deduplication operation so as to obtain target batch data after the data deduplication operation.
Exemplary, check and preprocessing of data, mainly checking the data format and the number of fields; such as comma separation, whether comma formats of each row are consistent; there is further a wrap in each field, if any.
Illustratively, the processed full and batch data is received into the central database.
Illustratively, the non-real-time data exchange method includes: an exchange standard is formulated, and exchange flow and specification of target equipment 6 are agreed; providing a data exchange channel for the source device 4 and the target device 6, and realizing centralized receiving and distributing; and providing data quality detection and data arrangement services of a certain program.
Specifically, the processed full data are combined, deduplicated, loaded in a trial mode, and finally received into a central database; and carrying out trial loading on the processed batch data, and finally, collecting the batch data into a central database.
Specifically, the step S104 may further include:
step S104B1, monitoring the processed process of the full data or batch data;
Step S104B2, when it is monitored that a field information modification event occurs in the processed procedure: generating warning information and writing the warning information into a modification log component, wherein the modification log component is used for informing the source equipment 4 that the initial data is modified, and the warning information comprises detailed information before modification and detailed information after modification;
Wherein the field information modification event includes a data modification event and a purge event including a purge event for a field exceeding a preset length and a purge event for data including a preset character.
Illustratively, when the system fails, a mechanism for manual intervention and repeatedly executed functions are provided for the system to continue to operate stably;
For example, when a cleaning event exceeding a field with a preset length occurs in the data preloading, the state of the state table of the data processing batch in the database is set to be failed, the failed batch is monitored by the monitoring system, log information is searched according to the batch, then the compensation running process is performed, meanwhile, the source device 4 is notified of the failure of data release by the mail system, the source device 4 automatically checks self-checking, manually collates the data, and releases the data according to the flow again.
Illustratively, when the field information modification event is detected, a warning message (warning) is generated, which includes the following specific steps: format conversion: the character set is processed in a unified way, and no Warning prompt is generated if the data is not modified; data merging: combining the files, and generating no Warning prompt if the data is not modified; data deduplication: performing de-duplication operation on the repeated record, and generating no Waring prompt if the data is not modified; data cleaning: cleaning the ultralong field and the data containing special characters, when the data is modified, a Warning prompt is needed, and detailed information before and after modification is written into the modification log component.
Specifically, the step S104 may further include:
Step S104B3, when it is detected that a processing failure event occurs in the processed procedure: suspending the data processing operation; generating data processing suspension operation information, and writing the data processing suspension operation information into a failure log component, wherein the failure log component is used for informing the source equipment 4 that the initial data uploading fails;
Wherein the data processing operation suspension information comprises reason description information triggering the data processing operation suspension; the processing failure event includes: overload event caused by data preloading exceeding a preset loading range.
Illustratively, when the processing failure event is detected, the processing of the full or batch data is aborted, including the steps of: verifying the file without error, taking out the generated file, and not performing Reject processing on the data; verification is incorrect, but the number of errors is in an acceptable range, and the data can be processed, so that Reject processing is not carried out on the data; if the verification is wrong and the data exceeds the processing range, ending the flow, submitting the self-checking Reject log, and writing into a failure log component.
Illustratively, data is stored in a temporary table, data is inserted into the temporary table, and whether the data is inserted is verified to be wrong, such as whether a verification field is too long, whether a data type is wrong, and the like; for example: when the inserted data are verified to have more than 5 problems, other data can be submitted first, and then abnormal data are processed independently; after verification, the intermediate table is cleaned. Therefore, the stable operation of the system can be ensured, the unified Reject and Warning mechanism is perfected, data which cannot be processed are submitted in a Reject mode, data which are automatically modified by the field system are submitted in a Warning mode, and the complete traceability of the data in the processing process is ensured.
Step S106, publishing the target data to the target device 6 through the publishing directory.
In an exemplary embodiment, the step S106 may further include:
Step S106A, when the target data is detected to be loaded into the release catalog for release, generating a notification release completion file and a notification acquisition data file;
Step S106B, the notification release completion file is sent to the source equipment 4, and the notification release completion file is used for notifying the source equipment 4 that release is completed; and
Step S106C of transmitting the notification acquisition data file to the target device 6 and notifying the target device 6 of acquisition data at a predetermined frequency.
Example two
Fig. 3 is a schematic diagram of a program module of a non-real-time data exchange system according to a second embodiment of the present invention. The non-real-time data exchange system 20 may include or be partitioned into one or more program modules that are stored in a storage medium and executed by one or more processors to perform the present invention and to implement the non-real-time data exchange methods described above. Program modules in accordance with the embodiments of the present invention are directed to a series of computer program instruction segments capable of performing the specified functions and are more suitable than the programs themselves for describing the execution of the non-real time data exchange system 20 in a storage medium. The following description will specifically describe functions of each program module of the present embodiment:
the obtaining module 200 is configured to obtain initial data from the source device 4, where the initial data is pre-labeled and classified data.
A loading module 202, configured to load the initial data into a receiving directory.
And the processing module 204 is used for carrying out data processing on the initial data in the receiving directory to obtain target data, and loading the target data into the release directory.
And the publishing module 206 is used for publishing the target data to the target device through the publishing catalog.
By way of example, the initial data may include vehicle insurance underwriting data, vehicle insurance claim data, personal insurance data, and reinsurance data.
Illustratively, the initial data includes a signal file (.ok or.ok.ing), a system number, a batch file list. The initial data of the same batch is the front-back data dependency relationship; the initial data is divided into batch data when the initial data has a dependency relationship, the batch data can be serially executed through a receiving-processing-trial loading-publishing sequence, and processes of the serial execution can be parallel; if the initial data has no dependency relationship, the initial data is divided into full data, processes can be executed in parallel, the maximum process number can be set in configuration, and the concurrency can be controlled.
Illustratively, the initial data includes full-volume data and/or batch data, and the receiving catalog includes a near real-time receiving catalog and a batch receiving catalog; loading the initial data into a receiving directory; the loading module 202 is further configured to: judging whether a demand instruction of the target equipment 6 is received or not, wherein the demand instruction comprises a full-increment demand instruction; if the demand instruction is received, correspondingly loading the full data into the quasi-real-time receiving catalog, and correspondingly loading the batch data into the batch receiving catalog; and if the demand instruction is not received, loading the full data and the batch data into a batch receiving catalog.
Illustratively, for the target device 6, data loading does not distinguish between bulk and full, both are deblocked in bulk; if the target device 6 has a full increment requirement, the full increment logic needs to be implemented in the increment data file of the source device 4; the data is generally provided in a full-scale manner as a basic configuration table with very small data volume and little variation.
Illustratively, the processing module 204 is further configured to: if the demand instruction is received, correspondingly issuing the target full-volume data to the quasi-real-time issuing catalog, and correspondingly issuing the target batch data to the increment issuing catalog; and if the demand instruction is not received, the target full-volume data and the batch data are issued to an increment issue catalog.
Illustratively, the processing module 204 is further configured to: monitoring the processed process of the full data or batch data; when the field information modification event occurs in the processed process, the processing method comprises the following steps: generating warning information and writing the warning information into a modification log component, wherein the log component is used for informing the source equipment 4 that the initial data is modified, and the warning information comprises detailed information before modification and detailed information after modification; wherein the field information modification event includes a data modification event and a purge event including a purge event for a field exceeding a preset length and a purge event for data including a preset character.
Illustratively, the processing module 204 is further configured to: when the processing failure event occurs in the processed process is monitored: suspending the data processing operation; generating data processing suspension operation information, and writing the data processing suspension operation information into a failure log component, wherein the failure log component is used for informing the source equipment 4 that the initial data uploading fails; wherein the data processing operation suspension information comprises reason description information triggering the data processing operation suspension; the processing failure event includes: overload event caused by data preloading exceeding a preset loading range.
Illustratively, the publishing module 206 is further configured to: when the target data is detected to be loaded into the release catalog for release, generating a notification release completion file and a notification acquisition data file; the notification release completion file is sent to source equipment and used for notifying the source equipment that release is completed; and sending the notification acquisition data file to target equipment, and notifying the target equipment to acquire target data at a preset frequency.
Example III
Referring to fig. 4, a hardware architecture diagram of a computer device according to a third embodiment of the present invention is shown. In this embodiment, the computer device 2 is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction. The computer device 2 may be a rack server, a blade server, a tower server, or a rack server (including a stand-alone server, or a server cluster made up of multiple servers), or the like. As shown, the computer device 2 includes, but is not limited to, at least a memory 21, a processor 22, a network interface 23, and a non-real time data exchange system 20 communicatively coupled to each other via a system bus.
In this embodiment, the memory 21 includes at least one type of computer-readable storage medium including flash memory, a hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 21 may be an internal storage unit of the computer device 2, such as a hard disk or a memory of the computer device 2. In other embodiments, the memory 21 may also be an external storage device of the computer device 2, such as a plug-in hard disk provided on the computer device 2, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), or the like. Of course, the memory 21 may also include both internal storage units of the computer device 2 and external storage devices. In this embodiment, the memory 21 is typically used to store an operating system and various types of application software installed on the computer device 2, such as program codes of the non-real-time data exchange system 20 of the second embodiment. Further, the memory 21 may be used to temporarily store various types of data that have been output or are to be output.
Processor 22 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 22 is typically used to control the overall operation of the computer device 2. In this embodiment, the processor 22 is configured to execute the program code stored in the memory 21 or process data, for example, execute the non-real-time data exchange system 20, so as to implement the non-real-time data exchange method of the first embodiment.
The network interface 23 may comprise a wireless network interface or a wired network interface, which network interface 23 is typically used for establishing a communication connection between the computer apparatus 2 and other electronic devices. For example, the network interface 23 is used to connect the computer device 2 to an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 2 and the external terminal, and the like. The network may be an Intranet (Intranet), the Internet (Internet), a global system for mobile communications (Global System of Mobile communication, GSM), wideband code division multiple access (Wideband Code Division Multiple Access, WCDMA), a 4G network, a 5G network, bluetooth (Bluetooth), wi-Fi, or other wireless or wired network.
It is noted that fig. 4 only shows a computer device 2 having components 20-23, but it is understood that not all of the illustrated components are required to be implemented, and that more or fewer components may alternatively be implemented.
In the present embodiment, the non-real-time data exchange system 20 stored in the memory 21 may also be divided into one or more program modules, which are stored in the memory 21 and executed by one or more processors (the processor 22 in the present embodiment) to complete the present invention.
For example, fig. 3 shows a schematic diagram of a program module for implementing the non-real-time data exchange system 20 according to the second embodiment of the present invention, where the non-real-time data exchange system 20 may be divided into an acquisition module 200, a loading module 202, a processing module 204, and a publishing module 206. Program modules in the present invention are understood to mean a series of computer program instruction segments capable of performing a specific function, more appropriately than a program, describing the execution of the non-real time data exchange system 20 in the computer device 2. The specific functions of the program modules 200-206 are described in detail in the second embodiment, and are not described herein.
Example IV
The present embodiment also provides a computer-readable storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, which when executed by a processor, performs the corresponding functions. The computer readable storage medium of the present embodiment is used in the non-real time data exchange system 20, and when executed by a processor, implements the non-real time data exchange method of the first embodiment.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (7)
1. A method of non-real time data exchange, the method comprising:
acquiring initial data from source equipment, wherein the initial data are pre-marked and classified data;
Loading the initial data into a receiving directory;
Performing data processing on the initial data in the receiving catalog to obtain target data, and loading the target data into the release catalog; and
Publishing the target data to target equipment through a publishing directory;
The receiving catalogs comprise a quasi-real-time receiving catalogs and a batch receiving catalogs, and the release catalogs comprise a quasi-real-time release catalogs and an increment release catalogs;
the quasi real-time receiving catalogue is connected with the quasi real-time publishing catalogue through a quasi real-time exchange module;
the batch receiving catalogue is connected with the increment release catalogue through a batch exchange module;
The initial data comprise full data or/and batch data, and the receiving catalogue comprises a quasi-real-time receiving catalogue and a batch receiving catalogue; loading the initial data into a receiving directory, comprising:
judging whether a demand instruction of target equipment is received or not, wherein the demand instruction comprises a full-increment demand instruction;
if the demand instruction is received, correspondingly loading the full data into the quasi-real-time receiving catalog, and correspondingly loading the batch data into the batch receiving catalog; and
If the demand instruction is not received, loading the full data and the batch data into a batch receiving catalog;
The target data comprise target full-volume data or/and target batch data, and the release catalogue comprises a quasi-real-time release catalogue and an incremental release catalogue; loading the target data into a release catalog, including:
if the demand instruction is received, correspondingly issuing the target full-volume data to the quasi-real-time issuing catalog, and correspondingly issuing the target batch data to the increment issuing catalog;
And if the demand instruction is not received, the target full-volume data and the batch data are issued to an increment issue catalog.
2. The non-real time data exchange method according to claim 1, wherein said data processing the initial data in the receiving directory to obtain the target data comprises:
monitoring the processed process of the full data or batch data;
When the field information modification event occurs in the processed process, the processing method comprises the following steps: generating alert information and writing the alert information into a modification log component, wherein the modification log component is used for informing the source device that the initial data has been modified, and the alert information comprises detailed information before modification and detailed information after modification;
Wherein the field information modification event includes a data modification event and a cleaning event including a cleaning event for a field exceeding a preset length and a cleaning event for data including a preset character.
3. The non-real time data exchange method according to claim 2, wherein said data processing the initial data in the receiving directory to obtain the target data further comprises:
when the processing failure event occurs in the processed process is monitored: suspending the data processing operation; generating data processing suspending operation information, writing the data processing suspending operation information into a failure log component, wherein the failure log component is used for informing the source equipment that the initial data uploading fails;
Wherein the data processing operation suspension information comprises reason description information triggering the data processing operation suspension; the processing failure event includes: overload event caused by data preloading exceeding a preset loading range.
4. The non-real time data exchange method according to claim 3, wherein publishing the target data to a target device through a publishing directory, further comprising:
when the target data is detected to be loaded into the release catalog for release, generating a notification release completion file and a notification acquisition data file;
The notification release completion file is sent to source equipment and used for notifying the source equipment that release is completed; and
And sending the notification acquisition data file to the target equipment, and notifying the target equipment to acquire the target data at a preset frequency.
5. A non-real-time data exchange system, comprising:
The acquisition module is used for acquiring initial data from the source equipment, wherein the initial data are pre-marked and classified data;
the loading module is used for loading the initial data to a receiving catalog;
the processing module is used for carrying out data processing on the initial data in the receiving catalog to obtain target data, and loading the target data into the release catalog; and
The publishing module is used for publishing the target data to the target equipment through a publishing catalog;
The receiving catalogs comprise a quasi-real-time receiving catalogs and a batch receiving catalogs, and the release catalogs comprise a quasi-real-time release catalogs and an increment release catalogs;
the quasi real-time receiving catalogue is connected with the quasi real-time publishing catalogue through a quasi real-time exchange module;
the batch receiving catalogue is connected with the increment release catalogue through a batch exchange module;
The initial data comprise full data or/and batch data, and the receiving catalogue comprises a quasi-real-time receiving catalogue and a batch receiving catalogue; the loading module is further configured to:
judging whether a demand instruction of target equipment is received or not, wherein the demand instruction comprises a full-increment demand instruction;
if the demand instruction is received, correspondingly loading the full data into the quasi-real-time receiving catalog, and correspondingly loading the batch data into the batch receiving catalog; and
If the demand instruction is not received, loading the full data and the batch data into a batch receiving catalog;
the target data comprise target full-volume data or/and target batch data, and the release catalogue comprises a quasi-real-time release catalogue and an incremental release catalogue; the publishing module is further configured to:
if the demand instruction is received, correspondingly issuing the target full-volume data to the quasi-real-time issuing catalog, and correspondingly issuing the target batch data to the increment issuing catalog;
And if the demand instruction is not received, the target full-volume data and the batch data are issued to an increment issue catalog.
6. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the computer program when executed by the processor implements the steps of the non-real time data exchange method according to any of claims 1 to 4.
7. A computer readable storage medium, characterized in that it has stored therein a computer program executable by at least one processor to cause the at least one processor to perform the steps of the non-real time data exchange method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910968954.4A CN110865990B (en) | 2019-10-12 | 2019-10-12 | Non-real-time data exchange method, system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910968954.4A CN110865990B (en) | 2019-10-12 | 2019-10-12 | Non-real-time data exchange method, system and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110865990A CN110865990A (en) | 2020-03-06 |
| CN110865990B true CN110865990B (en) | 2024-04-30 |
Family
ID=69652674
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910968954.4A Active CN110865990B (en) | 2019-10-12 | 2019-10-12 | Non-real-time data exchange method, system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110865990B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2545232A1 (en) * | 2005-07-29 | 2007-01-29 | Cognos Incorporated | Method and system for creating a taxonomy from business-oriented metadata content |
| JP2007213209A (en) * | 2006-02-08 | 2007-08-23 | Sharp Corp | Data management device, data storage, data management method, program, and recording medium |
| CN105183391A (en) * | 2015-09-18 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Method and device for storing data under distributed data platform |
| CN108829767A (en) * | 2018-05-29 | 2018-11-16 | 吉贝克信息技术（北京）有限公司 | Data exchange system and its method, apparatus and computer storage medium |
| CN109344123A (en) * | 2018-10-23 | 2019-02-15 | 山东浪潮云信息技术有限公司 | A kind of data interactive method based on catalogue file |
-
2019
- 2019-10-12 CN CN201910968954.4A patent/CN110865990B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2545232A1 (en) * | 2005-07-29 | 2007-01-29 | Cognos Incorporated | Method and system for creating a taxonomy from business-oriented metadata content |
| JP2007213209A (en) * | 2006-02-08 | 2007-08-23 | Sharp Corp | Data management device, data storage, data management method, program, and recording medium |
| CN105183391A (en) * | 2015-09-18 | 2015-12-23 | 北京京东尚科信息技术有限公司 | Method and device for storing data under distributed data platform |
| CN108829767A (en) * | 2018-05-29 | 2018-11-16 | 吉贝克信息技术（北京）有限公司 | Data exchange system and its method, apparatus and computer storage medium |
| CN109344123A (en) * | 2018-10-23 | 2019-02-15 | 山东浪潮云信息技术有限公司 | A kind of data interactive method based on catalogue file |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110865990A (en) | 2020-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109032824B (en) | Database verification method, database verification device, computer equipment and storage medium | |
| CN110599354B (en) | Online checking method, online checking system, computer device and computer readable storage medium | |
| CN110222535B (en) | Processing device, method and storage medium for block chain configuration file | |
| CN109951514B (en) | File processing method and system based on cloud storage and computer equipment | |
| CN112527484B (en) | Workflow breakpoint continuous running method and device, computer equipment and readable storage medium | |
| CN108170551B (en) | Crawler system based front-end and back-end error processing method, server and storage medium | |
| CN109491819A (en) | A kind of method and system of diagnosis server failure | |
| CN102479113A (en) | Abnormal self-adapting processing method and system | |
| CN110866834B (en) | Execution method and system of batch processing program | |
| CN110958292A (en) | File uploading method, electronic device, computer equipment and storage medium | |
| CN112561370A (en) | Software version management method and device, computer equipment and storage medium | |
| CN111858605A (en) | Database automatic auditing method, system, equipment and storage medium | |
| CN111367982A (en) | Method and device for importing TRIRIGA basic data, computer equipment and storage medium | |
| CN110865990B (en) | Non-real-time data exchange method, system and computer equipment | |
| CN110069382B (en) | Software monitoring method, server, terminal device, computer device and medium | |
| CN112181836A (en) | Test case generation method, system, device and storage medium | |
| CN117474696A (en) | Diagnosis method, system, equipment and storage medium for commission settlement problem | |
| CN110704462A (en) | Data change notification method, electronic device, computer device, and storage medium | |
| CN112256532A (en) | Test interface generation method and device, computer equipment and readable storage medium | |
| CN112395266B (en) | Data file processing method, first server and computer storage medium | |
| CN113242147B (en) | Automatic operation and maintenance deployment method, device, equipment and storage medium of multi-cloud environment | |
| CN114168108A (en) | Code merging method and device, electronic equipment and computer readable storage medium | |
| CN109828884B (en) | Hanging service data processing method, system, computer equipment and storage medium | |
| CN112558982A (en) | Code detection method and device and computer equipment | |
| CN113051165A (en) | Method, device, monitoring server and medium for processing test order |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |