CN116662263A - Data receiving and processing method and device and computer equipment - Google Patents
Data receiving and processing method and device and computer equipment Download PDFInfo
- Publication number
- CN116662263A CN116662263A CN202310634457.7A CN202310634457A CN116662263A CN 116662263 A CN116662263 A CN 116662263A CN 202310634457 A CN202310634457 A CN 202310634457A CN 116662263 A CN116662263 A CN 116662263A
- Authority
- CN
- China
- Prior art keywords
- data
- information
- processing
- data file
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 520
- 238000000034 method Methods 0.000 claims abstract description 38
- 238000012795 verification Methods 0.000 claims description 52
- 238000004590 computer program Methods 0.000 claims description 47
- 238000007781 pre-processing Methods 0.000 claims description 21
- 238000012216 screening Methods 0.000 claims description 17
- 238000012546 transfer Methods 0.000 claims description 13
- 238000012163 sequencing technique Methods 0.000 claims description 11
- 230000008569 process Effects 0.000 abstract description 17
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 4
- 230000006837 decompression Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0894—Policy-based network configuration management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/122—File system administration, e.g. details of archiving or snapshots using management policies
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data receiving and processing method, a device and computer equipment, belonging to the technical field of big data. The method comprises the following steps: receiving information of a data file to be processed; the data file information to be processed comprises data file identification information; determining a data processing link of a target type matched with the identification information of the data file from the data processing links of all preset types as a target data processing link; and processing the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link. The method can effectively determine the target data processing link corresponding to the data information of the file to be processed, and further efficiently and accurately process the data information of the file to be processed by utilizing the processing strategy corresponding to the target data processing link, thereby improving the data receiving and processing efficiency.
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a data receiving and processing method, an apparatus, and a computer device.
Background
Currently, asset hosting systems access daily transaction content from up-hand offices, deep-hand offices and other various types of exchanges in the form of data reception. As the size of hosted services has gradually increased, both the amount of data and the type of instructions that enter the system through data reception have continually expanded.
However, once an extreme situation occurs, such as that the service concentrates in a certain period to submit a large number of instructions, the sequence of data reception is often accumulated, the processing speed is slow, and the instructions needing to be processed preferentially are accumulated, so that timely processing cannot be performed.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data reception processing method, apparatus, and computer device capable of efficiently performing data processing.
In a first aspect, the present application provides a data receiving processing method. The method comprises the following steps:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file from the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link.
In one embodiment, the processing the information of the data file to be processed through the target data processing link and the data processing policy corresponding to the target data processing link includes:
decompressing the data file information to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
and when the data file identification information is the target identification information, screening out the target data result information based on the data screening condition in the data result information.
In one embodiment, the data receiving and processing method further includes:
and when the data file identification information does not meet a preset identification condition, taking all the data processing links of the preset type as the target data processing links.
In one embodiment, the data receiving and processing method further includes:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to all the data processing links of the preset type to obtain data processing result information corresponding to each data processing link of the preset type;
And verifying the data processing result information corresponding to each data processing link of the preset type according to the target data result information of the verification data file information, and generating verification information.
In one embodiment, the data receiving and processing method further includes:
matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment, the processing the initial data file information according to the data preprocessing policy to obtain the data file information to be processed includes:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment, the data receiving and processing method further includes:
sequencing according to the processing priority corresponding to each data processing link of the preset type from high to low to obtain a processing mode sequence;
And under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
In a second aspect, the application further provides a data receiving and processing device. The device comprises:
the receiving module is used for receiving the information of the data file to be processed; the data file information to be processed comprises data file identification information;
the matching module is used for determining a data processing link of a target type matched with the data file identification information in the data processing links of all preset types as a target data processing link;
and the processing module is used for processing the data file information to be processed through the target data processing link and the data processing strategy corresponding to the target data processing link.
In one embodiment, the processing module is specifically configured to:
decompressing the data file information to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
And when the data file identification information is the target identification information, screening out the target data result information based on the data screening condition in the data result information.
In one embodiment, the data receiving and processing device further includes:
and the identification module is used for taking all the data processing links of the preset type as the target data processing links when the data file identification information does not meet the preset identification condition.
In one embodiment, the data receiving and processing device further includes a verification module, where the verification module is specifically configured to:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to all the data processing links of the preset type to obtain data processing result information corresponding to each data processing link of the preset type;
and verifying the data processing result information corresponding to each data processing link of the preset type according to the target data result information of the verification data file information, and generating verification information.
In one embodiment, the data receiving and processing device further includes a to-be-processed data file information determining module, where the to-be-processed data file information determining module is specifically configured to:
matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment, the to-be-processed data file information determining module is specifically configured to:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment, the data receiving and processing device further includes a target data processing link determining module, where the target data processing link determining module is specifically configured to:
sequencing according to the processing priority corresponding to each data processing link of the preset type from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file from the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link.
In one embodiment, the processing the information of the data file to be processed through the target data processing link and the data processing policy corresponding to the target data processing link includes:
decompressing the data file information to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
and when the data file identification information is the target identification information, screening out the target data result information based on the data screening condition in the data result information.
In one embodiment, the processor when executing the computer program further performs the steps of:
and when the data file identification information does not meet a preset identification condition, taking all the data processing links of the preset type as the target data processing links.
In one embodiment, the processor when executing the computer program further performs the steps of:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to all the data processing links of the preset type to obtain data processing result information corresponding to each data processing link of the preset type;
and verifying the data processing result information corresponding to each data processing link of the preset type according to the target data result information of the verification data file information, and generating verification information.
In one embodiment, the processor when executing the computer program further performs the steps of:
Matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment, the processing the initial data file information according to the data preprocessing policy to obtain the data file information to be processed includes:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment, the processor when executing the computer program further performs the steps of:
sequencing according to the processing priority corresponding to each data processing link of the preset type from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file from the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link.
In one embodiment, the processing the information of the data file to be processed through the target data processing link and the data processing policy corresponding to the target data processing link includes:
decompressing the data file information to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
and when the data file identification information is the target identification information, screening out the target data result information based on the data screening condition in the data result information.
In one embodiment, the computer program when executed by a processor further performs the steps of:
and when the data file identification information does not meet a preset identification condition, taking all the data processing links of the preset type as the target data processing links.
In one embodiment, the computer program when executed by a processor further performs the steps of:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to all the data processing links of the preset type to obtain data processing result information corresponding to each data processing link of the preset type;
and verifying the data processing result information corresponding to each data processing link of the preset type according to the target data result information of the verification data file information, and generating verification information.
In one embodiment, the computer program when executed by a processor further performs the steps of:
Matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment, the processing the initial data file information according to the data preprocessing policy to obtain the data file information to be processed includes:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment, the computer program when executed by a processor further performs the steps of:
sequencing according to the processing priority corresponding to each data processing link of the preset type from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
In a fifth aspect, the present application also provides a computer program product. The computer program product comprising a computer program which, when executed by a processor, performs the steps of:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file from the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link.
In one embodiment, the processing the information of the data file to be processed through the target data processing link and the data processing policy corresponding to the target data processing link includes:
decompressing the data file information to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
and when the data file identification information is the target identification information, screening out the target data result information based on the data screening condition in the data result information.
In one embodiment, the computer program when executed by a processor further performs the steps of:
and when the data file identification information does not meet a preset identification condition, taking all the data processing links of the preset type as the target data processing links.
In one embodiment, the computer program when executed by a processor further performs the steps of:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to all the data processing links of the preset type to obtain data processing result information corresponding to each data processing link of the preset type;
and verifying the data processing result information corresponding to each data processing link of the preset type according to the target data result information of the verification data file information, and generating verification information.
In one embodiment, the computer program when executed by a processor further performs the steps of:
Matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment, the processing the initial data file information according to the data preprocessing policy to obtain the data file information to be processed includes:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment, the computer program when executed by a processor further performs the steps of:
sequencing according to the processing priority corresponding to each data processing link of the preset type from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
According to the data receiving and processing method, the device and the computer equipment, the data file identification information of the data file information to be processed is matched with the data processing links of all preset types, the target data processing link matched with the data file identification information is determined, and the data file information to be processed is processed according to the target data processing link and the corresponding data processing strategy. The method can effectively determine the target data processing link corresponding to the data information of the file to be processed, and further efficiently and accurately process the data information of the file to be processed by utilizing the processing strategy corresponding to the target data processing link, thereby improving the data receiving and processing efficiency.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments or the conventional techniques of the present application, the drawings required for the descriptions of the embodiments or the conventional techniques will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present application, and other drawings may be obtained according to the drawings without inventive effort for those skilled in the art.
FIG. 1 is a flow chart of a data receiving and processing method in one embodiment;
FIG. 2 is a flow chart of a data receiving and processing method according to another embodiment;
FIG. 3 is a flow chart of a data receiving and processing method according to another embodiment;
FIG. 4 is a block diagram of a data receiving and processing system in one embodiment
FIG. 5 is a block diagram showing the structure of a data receiving processing apparatus in one embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
In one embodiment of the present application, as shown in fig. 1, a data receiving and processing method is provided, where this embodiment is exemplified by the method applied to a terminal (may be referred to as a management terminal), and it is understood that the method may also be applied to a server, and may also be applied to a system including a terminal and a server, and implemented through interaction between the terminal and the server. The terminal may be, but not limited to, various personal computers and notebook computers. The server may be implemented as a stand-alone server or as a server cluster composed of a plurality of servers. In this embodiment, the method includes the steps of:
Step 101, receiving information of a data file to be processed; the data file information to be processed includes data file identification information.
The terminal can receive the data file information to be processed transmitted by the transfer machine, and the data file identification information can identify the type of the data file information to be processed.
Step 102, determining a data processing link of a target type matched with the identification information of the data file from the data processing links of each preset type as a target data processing link.
The terminal is used for matching the data file identification information in the data file information to be processed with the data processing link identification information corresponding to the data processing links of each preset type, and the data processing link corresponding to the data processing link identification information successfully matched with the data file identification information is used as a target data processing link corresponding to the data file information to be processed.
And step 103, processing the information of the data file to be processed through the target data processing link and the data processing strategy corresponding to the target data processing link.
The terminal stores data processing strategies corresponding to the data processing links, and further the terminal can process the data file information to be processed according to the target data processing links and the data processing strategies corresponding to the target data processing links.
In the data receiving and processing method, the data file identification information of the data file information to be processed is matched with the data processing links of each preset type, the target data processing link matched with the data file identification information is determined, and the data file information to be processed is processed according to the target data processing link and the corresponding data processing strategy. The data receiving processing method can determine the data processing link matched with the information of the data file to be processed, and further can accurately process the information of the data file to be processed according to the data processing strategy of the matched data processing link, so that the data receiving processing efficiency is improved, and meanwhile, the accuracy of the data receiving processing is also improved.
In one embodiment of the present application, the step 103 processes the information of the data file to be processed through the target data processing link and the data processing policy corresponding to the target data processing link, including:
Decompressing the information of the data file to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
when the data file identification information is target identification information, the target data result information is screened out based on the data screening condition in the data result information.
Specifically, the terminal obtains the information of the data file to be processed from the shared storage, and it should be noted that the information of the data file to be processed is stored in the compressed package form and the shared storage. Based on the above, the terminal needs to decompress the acquired information of the data file to be processed, and the decompression mode can be selected according to actual requirements, including but not limited to decompression based on java programs, which is not limited in the application. Secondly, the terminal needs to utilize a database processing tool to process the decompressed data file information to be processed, generates data result information and stores the data result information in a database, wherein the data result information comprises data information of each link corresponding to the decompressed data information. For example, the terminal may process the decompressed data information by using an sqlldr (a database processing tool), generate a corresponding ctl file, and drop the corresponding data into an oracle database.
It should be noted that when the data file identification information is the target identification information, after obtaining the data result information corresponding to the data information to be processed, the target data result information needs to be screened out from the data result information, and the target data result information is stored in the database. In one embodiment of the present application, the data filtering condition may be that data result information corresponding to a last processing link in the data processing links is stored in the database as target data result information.
In summary, when the data file identification information is the target identification information, the target data result information is screened out from the data result information, and the target data result information is stored in the database, and when the data file identification information is not the target identification information, the data result information is used as the target data result information, and the obtained data result information is directly stored in the database. In one embodiment of the present application, the data file information to be processed corresponding to the target identification information may be high-sensitivity instruction data file information. When the data receiving peak period occurs, the information of the high-sensitivity instruction data file enters the high-sensitivity link for processing, so that the circulation speed of the high-sensitivity instruction is greatly accelerated, and the processing timeliness is ensured.
In the data receiving and processing method, different data processing is carried out on the data file information to be processed according to whether the data file identification information of the data file information to be processed is the target identification information, and the corresponding target data result information is determined, so that the method is beneficial to improving the efficiency of information processing of various types of data files to be processed and also can improve the accuracy of the corresponding target data result information.
In one embodiment of the present application, the data reception processing method further includes:
and when the data file identification information does not meet the preset identification condition, taking all the data processing links of the preset type as target data processing links.
Specifically, the terminal stores preset data processing types, matches data file identification information of data file information to be processed with the preset data processing types, uses the data file information to be processed, which is failed to match, as verification data file information, uses all data processing links of the preset types as target data processing links, and processes the data processing links of the preset types.
In one embodiment of the present application, as shown in fig. 2, the data receiving processing method further includes:
In step 201, when the data file identification information does not meet the preset identification condition, the data file information to be processed is used as verification data file information.
The terminal stores preset data processing types, matches the data file identification information of the data file information to be processed with the preset data processing types, uses the data file information to be processed, which is failed to match, as verification data file information, uses all the data processing links of the preset types as target data processing links, and processes the data processing links of the preset types.
Step 202, processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to the data processing links of the preset type, and obtaining data processing result information corresponding to the data processing links of the preset type.
The terminal processes the verification data file information according to the data processing links and the corresponding data processing strategies thereof for each preset type of data processing link to obtain data processing result information corresponding to each preset type of data processing link.
Step 203, verifying the data processing result information corresponding to each preset type of data processing link according to the target data result information of the verification data file information, and generating verification information.
Specifically, the terminal stores target data result information corresponding to the verification data file information, so that each preset type of data processing link can be verified according to the target data result information. When the target data result information is consistent with the data processing result information of the data processing link, verifying that the data processing link is correct, and if the target data result information is inconsistent with the data processing result information of the data processing link, generating risk information by the terminal according to the inconsistent information so as to prompt related personnel to detect the data processing link.
In the data receiving and processing method, the verification data file information is processed based on the data processing links of each type and the strategy thereof, and the data processing links of each preset type are verified according to the target data processing result of the verification data file information and the data processing result information of the data processing links of each preset type, so that the correctness of the data processing links of each preset type can be effectively detected, and the accuracy of the data receiving and processing method is improved.
In one embodiment of the present application, as shown in fig. 3, the data receiving processing method further includes:
step 301, matching the received initial data file information and target data file directory information through the transfer machine.
As shown in fig. 4, the front end processor may receive the initial data file information transmitted through each service and transmit the initial data file information to the repeater. The receiver stores target data file directory information, and the receiver can judge whether target file information in the target data file directory information is received or not by matching the received initial data file information with the target data file directory information. The transfer machine can also judge whether the target file is received or not according to the target data file directory information and the received initial data file information in a timing operation mode.
Step 302, under the condition that the matching is successful, the initial data file information is processed according to the data preprocessing strategy, and the data file information to be processed is obtained.
Under the condition that the matching is successful, the receiver determines that the received initial data file information is the target file information. When the receiver needs to determine that the target file information is received, the initial data file information is downloaded to the local, and the initial data file information is processed according to the preprocessing strategy to obtain the data file information to be processed.
In one embodiment of the present application, the step 302 processes the initial data file information according to the data preprocessing policy to obtain the data file information to be processed, including:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
Specifically, after determining that the received initial data file information is the target file information, the relay device performs decompression backup processing on the initial data file information. Because the file formats of the initial data file information are different, the initial format of the initial data file information is required to be converted into a target file format, the initial data file information converted into the target file format is compressed to obtain the file information with the processed data, and the file information to be processed is transmitted to the shared storage for the subsequent work of the terminal.
As shown in fig. 4, the block diagram of the data receiving processing system includes a normal link and a high-sensitivity link, which correspond to the normal data file information and the high-sensitivity data file information, respectively. It should be noted that the number of links included in the data receiving and processing system is not limited to two, and can be set according to the requirement, and the present application is described by taking a normal link and a high-sensitivity link as examples only. When the terminal receives the data file information to be processed from the shared storage and recognizes that the data file identification information of the data file information to be processed is the common data identification information, the terminal processes the data file information to be processed based on the common link, decompresses the data file information respectively, processes the decompressed data information by using an sqlldr (a database processing tool), generates a corresponding ctl file, and drops the corresponding data into an oracle database (an old database in fig. 4). When the terminal recognizes that the data file identification information of the data file information to be processed is high-sensitivity data identification information (namely target data identification information), the terminal processes the data file information to be processed based on a common link, decompresses the data file information to be processed respectively, processes the decompressed data information by utilizing an sqlldr (a database processing tool), and stores the final data processing result information only in a new database by utilizing an instruction stream service. Before the high-sensitivity link is put into use, 5000 data files corresponding to instructions are received every day, the peak value of the queue can reach 1000 in the peak period of 9 points 30-10 points 30 and 15 points-17 points every day, and the average processing speed exceeds 10 minutes. After the high-sensitivity link is put into use, 5000 files per day are divided into 3500 normal and 1500 high-sensitivity files, and as high-sensitivity instructions are mostly generated in a peak period, the normal instructions are relatively scattered, the peak value of a queue in the peak period is not more than 400, and the average processing speed reaches 3 and half, wherein the high-sensitivity instruction speed is about 2 minutes, so that the original system processing pressure is greatly relieved.
In one embodiment of the present application, the data reception processing method further includes:
sequencing according to the processing priority corresponding to each preset type of data processing link from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
The terminal stores priority values corresponding to the data processing links, and the setting of the priority values can be set according to actual requirements, which is not limited in the application. The terminal can sort according to the priority values of the data processing links from the big to the small according to the priority values, and a processing mode sequence is obtained.
When determining a target data processing link, the terminal can detect the state of the target data processing link, and can process the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link under the condition of detecting that the target data processing link is correct; and under the condition of detecting the error of the target data processing link, taking the data processing link with the highest processing priority in the processing mode sequence after the target data processing link is removed as the target data processing link, detecting the state of the target data processing link, and repeating the steps until the condition of the target data processing link is detected to be correct, and processing the information of the data file to be processed.
In the data receiving and processing method, the switching of the target data processing link can be performed aiming at the situation of error of the target data processing link, so that the stability of the data receiving and processing method is improved, and the accuracy of the data processing method is also improved.
It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data receiving and processing device for realizing the above related data receiving and processing method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation in the embodiments of the data receiving processing apparatus or apparatuses provided below may refer to the limitation of the data receiving processing method hereinabove, and will not be repeated herein.
As shown in fig. 5, the present application further provides a data receiving and processing apparatus 500, which includes:
a receiving module 510, configured to receive information of a data file to be processed; the data file information to be processed comprises data file identification information;
a matching module 520, configured to determine, among the data processing links of each preset type, a data processing link of a target type that matches the data file identification information, as a target data processing link;
the processing module 530 is configured to process the information of the data file to be processed through the target data processing link and the data processing policy corresponding to the target data processing link.
In one embodiment of the present application, the processing module 530 is specifically configured to:
decompressing the information of the data file to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
when the data file identification information is target identification information, the target data result information is screened out based on the data screening condition in the data result information.
In one embodiment of the present application, the data reception processing apparatus further includes:
and the identification module is used for taking all the data processing links of the preset type as target data processing links when the data file identification information does not meet the preset identification conditions.
In one embodiment of the present application, the data receiving and processing apparatus further includes an authentication module, and the authentication module is specifically configured to:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to the data processing links of the preset type to obtain data processing result information corresponding to the data processing links of the preset type;
and verifying the data processing result information corresponding to each preset type of data processing link according to the target data result information of the verification data file information, and generating verification information.
In one embodiment of the present application, the data receiving and processing apparatus further includes a data file information determining module to be processed, and the data file information determining module to be processed is specifically configured to:
matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment of the present application, the data file information determining module to be processed is specifically configured to:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment of the present application, the data receiving and processing apparatus further includes a target data processing link determining module, and the target data processing link determining module is specifically configured to:
sequencing according to the processing priority corresponding to each preset type of data processing link from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
In one embodiment of the present application, a computer device, which may be a terminal, is provided, and an internal structure diagram thereof may be as shown in fig. 6. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data reception processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in FIG. 6 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
The application also provides computer equipment. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file in the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and the data processing strategy corresponding to the target data processing link.
In one embodiment of the present application, processing data file information to be processed through a target data processing link and a data processing policy corresponding to the target data processing link includes:
Decompressing the information of the data file to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
when the data file identification information is target identification information, the target data result information is screened out based on the data screening condition in the data result information.
In one embodiment of the application, the processor when executing the computer program further performs the steps of:
and when the data file identification information does not meet the preset identification condition, taking all the data processing links of the preset type as target data processing links.
In one embodiment of the application, the processor when executing the computer program further performs the steps of:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to the data processing links of the preset type to obtain data processing result information corresponding to the data processing links of the preset type;
and verifying the data processing result information corresponding to each preset type of data processing link according to the target data result information of the verification data file information, and generating verification information.
In one embodiment of the application, the processor when executing the computer program further performs the steps of:
matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment of the present application, processing initial data file information according to a data preprocessing policy to obtain data file information to be processed includes:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment of the application, the processor when executing the computer program further performs the steps of:
sequencing according to the processing priority corresponding to each preset type of data processing link from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
The application also provides a computer readable storage medium. A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file in the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and the data processing strategy corresponding to the target data processing link.
In one embodiment of the present application, processing data file information to be processed through a target data processing link and a data processing policy corresponding to the target data processing link includes:
decompressing the information of the data file to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
when the data file identification information is target identification information, the target data result information is screened out based on the data screening condition in the data result information.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
and when the data file identification information does not meet the preset identification condition, taking all the data processing links of the preset type as target data processing links.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to the data processing links of the preset type to obtain data processing result information corresponding to the data processing links of the preset type;
and verifying the data processing result information corresponding to each preset type of data processing link according to the target data result information of the verification data file information, and generating verification information.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
matching the received initial data file information with target data file directory information through a transfer machine;
And under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment of the present application, processing initial data file information according to a data preprocessing policy to obtain data file information to be processed includes:
converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
sequencing according to the processing priority corresponding to each preset type of data processing link from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
The application also provides a computer program product. Computer program product comprising a computer program which, when executed by a processor, realizes the steps of:
Receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file in the data processing links of all preset types as a target data processing link;
and processing the information of the data file to be processed through the target data processing link and the data processing strategy corresponding to the target data processing link.
In one embodiment of the present application, processing data file information to be processed through a target data processing link and a data processing policy corresponding to the target data processing link includes:
decompressing the information of the data file to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
when the data file identification information is target identification information, the target data result information is screened out based on the data screening condition in the data result information.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
and when the data file identification information does not meet the preset identification condition, taking all the data processing links of the preset type as target data processing links.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
when the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to the data processing links of the preset type to obtain data processing result information corresponding to the data processing links of the preset type;
and verifying the data processing result information corresponding to each preset type of data processing link according to the target data result information of the verification data file information, and generating verification information.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
In one embodiment of the present application, processing initial data file information according to a data preprocessing policy to obtain data file information to be processed includes:
Converting the initial file format of the initial data file information successfully matched into a target file format;
and compressing the initial data file information in the target file format to be used as the data file information to be processed.
In one embodiment of the application, the computer program when executed by the processor further implements the steps of:
sequencing according to the processing priority corresponding to each preset type of data processing link from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above-described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.
Claims (11)
1. A data reception processing method, the method comprising:
receiving information of a data file to be processed; the data file information to be processed comprises data file identification information;
determining a data processing link of a target type matched with the identification information of the data file from the data processing links of all preset types as a target data processing link;
And processing the information of the data file to be processed through the target data processing link and a data processing strategy corresponding to the target data processing link.
2. The method according to claim 1, wherein the processing the to-be-processed data file information by the target data processing link and the data processing policy corresponding to the target data processing link includes:
decompressing the data file information to be processed to obtain decompressed data information;
processing the decompressed data information according to a database processing tool to generate data result information;
and when the data file identification information is the target identification information, screening out the target data result information based on the data screening condition in the data result information.
3. The data reception processing method according to claim 1, characterized in that the method further comprises:
and when the data file identification information does not meet a preset identification condition, taking all the data processing links of the preset type as the target data processing links.
4. A data reception processing method according to claim 3, characterized in that the method further comprises:
When the data file identification information does not meet the preset identification condition, taking the data file information to be processed as verification data file information;
processing the verification data file information through all the data processing links of the preset type and the data processing strategies corresponding to all the data processing links of the preset type to obtain data processing result information corresponding to each data processing link of the preset type;
and verifying the data processing result information corresponding to each data processing link of the preset type according to the target data result information of the verification data file information, and generating verification information.
5. The data reception processing method according to claim 1, characterized in that the method further comprises:
matching the received initial data file information with target data file directory information through a transfer machine;
and under the condition of successful matching, processing the initial data file information according to a data preprocessing strategy to obtain the data file information to be processed.
6. The method according to claim 5, wherein the processing the initial data file information according to the data preprocessing policy to obtain the data file information to be processed includes:
Converting the initial file format of the initial data file information successfully matched into a target file format; and compressing the initial data file information in the target file format to be used as the data file information to be processed.
7. The data reception processing method according to claim 1, characterized in that the method further comprises:
sequencing according to the processing priority corresponding to each data processing link of the preset type from high to low to obtain a processing mode sequence;
and under the condition that the target data processing link is in error, eliminating the target data processing link in the processing mode sequence, and taking the data processing link with the highest processing priority in the processing mode sequence after eliminating the target data processing link as the target data processing link.
8. A data reception processing apparatus, characterized in that the apparatus comprises:
the receiving module is used for receiving the information of the data file to be processed; the data file information to be processed comprises data file identification information;
the matching module is used for determining a data processing link of a target type matched with the data file identification information in the data processing links of all preset types as a target data processing link;
And the processing module is used for processing the data file information to be processed through the target data processing link and the data processing strategy corresponding to the target data processing link.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
11. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310634457.7A CN116662263A (en) | 2023-05-31 | 2023-05-31 | Data receiving and processing method and device and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310634457.7A CN116662263A (en) | 2023-05-31 | 2023-05-31 | Data receiving and processing method and device and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116662263A true CN116662263A (en) | 2023-08-29 |
Family
ID=87720213
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310634457.7A Pending CN116662263A (en) | 2023-05-31 | 2023-05-31 | Data receiving and processing method and device and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116662263A (en) |
-
2023
- 2023-05-31 CN CN202310634457.7A patent/CN116662263A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11385878B2 (en) | Model deployment method, model deployment device and terminal equipment | |
| WO2022082891A1 (en) | Big data acquisition method and system, and computer device and storage medium thereof | |
| CN107133233B (en) | Processing method and device for configuration data query | |
| CN116049109A (en) | File verification method, system, equipment and medium based on filter | |
| CN112328631A (en) | Production fault analysis method and device, electronic equipment and storage medium | |
| CN106980618B (en) | File storage method and system based on MongoDB distributed cluster architecture | |
| CN110019347B (en) | Data processing method and device of block chain and terminal equipment | |
| CN116561003A (en) | Test data generation method, device, computer equipment and storage medium | |
| CN116662263A (en) | Data receiving and processing method and device and computer equipment | |
| CN110765131A (en) | Data compression method and device for goods source data, computer equipment and storage medium | |
| CN113535449B (en) | Abnormal event restoration processing method and device, computer equipment and storage medium | |
| CN107704557B (en) | Processing method and device for operating mutually exclusive data, computer equipment and storage medium | |
| CN113111078A (en) | Resource data processing method and device, computer equipment and storage medium | |
| CN114218056B (en) | Performance test method and device for transaction system, computer equipment and storage medium | |
| CN118096171A (en) | Data gray level verification method, device, computer equipment, medium and product | |
| CN117827814A (en) | Data verification method, device, computer equipment and storage medium | |
| CN116541137A (en) | Transaction processing method, apparatus, computer device, storage medium, and program product | |
| CN117151841A (en) | Service processing method, device, computer equipment, storage medium and program product | |
| CN117910035A (en) | Data import and export method, system, computer device and storage medium | |
| CN116881116A (en) | Interface test method, apparatus, computer device, storage medium, and program product | |
| CN117240578A (en) | Flow integrity verification method and device, computer equipment and storage medium thereof | |
| CN116894031A (en) | Data quality detection method and device and computer equipment | |
| CN115964188A (en) | Data processing method, system, device, computer equipment and storage medium | |
| CN116760826A (en) | Resource allocation method, apparatus, device, storage medium and program product | |
| CN116089296A (en) | Service logic testing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |