CN115017216A - Data processing method and related device - Google Patents
Data processing method and related device Download PDFInfo
- Publication number
- CN115017216A CN115017216A CN202210405525.8A CN202210405525A CN115017216A CN 115017216 A CN115017216 A CN 115017216A CN 202210405525 A CN202210405525 A CN 202210405525A CN 115017216 A CN115017216 A CN 115017216A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- data processing
- processed
- units
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
After a to-be-processed data set is obtained, a processing device may analyze a plurality of data types corresponding to data included in the to-be-processed data set, and then generate a plurality of data units corresponding to the to-be-processed data set, where the data types included in the same data unit are the same. Subsequently, the processing device may perform data processing on data included in the plurality of data units based on the data types respectively corresponding to the plurality of data units, thereby enabling efficient data processing for data including a plurality of data types.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method and a related apparatus.
Background
With the development of economy and science and technology, the business scene of the financial industry is more and more complex, the channel service diversification development, and in the framework transformation service optimization, the system platform is continuously extended and expanded, and more system technology platforms appear.
Different system technology platforms generally use different data types of data, and in the related art, a data processing tool can only process data according to data in a certain format. When the data format of the system is changed, the data loading tool needs to be changed, and the modification cost is high.
Disclosure of Invention
In order to solve the technical problem, the application provides a data processing method, and a processing device can process acquired diversified data more efficiently.
The embodiment of the application discloses the following technical scheme:
in a first aspect, an embodiment of the present application discloses a data processing method, where the method includes:
acquiring a data set to be processed, wherein the data set to be processed comprises data corresponding to a plurality of data types;
generating a plurality of data units corresponding to the data set to be processed, wherein the data types of the data included in the same data unit are the same;
and processing data included in the data units based on the data types respectively corresponding to the data units.
In a possible implementation manner, the performing data processing on the data included in the plurality of data units based on the data types respectively corresponding to the plurality of data units includes:
determining data processing programs respectively corresponding to the data units based on the data types respectively corresponding to the data units;
and processing the data in the data units according to the data processing program.
In one possible implementation, the data processing includes any one or combination of processing loading and data inspection.
In a possible implementation, before the acquiring the set of data to be processed, the method further includes;
and generating a data processing program corresponding to each data type aiming at each data type.
In a possible implementation manner, the generating a data processing program corresponding to each data type includes:
determining data operation parameters corresponding to the target data type, wherein the data operation parameters comprise any one or combination of multiple data operation scene parameters, data processing flow parameters and data processing parameters;
and generating a data processing program corresponding to the target data type according to the data operation parameters.
In one possible implementation, the method further includes:
acquiring a data processing result corresponding to the data processing;
and sending the data processing result to a responsible person corresponding to the data set to be processed.
In a second aspect, an embodiment of the present application discloses a data processing apparatus, which includes a first obtaining unit, a first generating unit, and a processing unit:
the first acquisition unit is used for acquiring a data set to be processed, and the data set to be processed comprises data corresponding to a plurality of data types;
the first generating unit is configured to generate a plurality of data units corresponding to the to-be-processed data set, where data types of data included in the same data unit are the same;
and the processing unit is used for carrying out data processing on the data included in the data units based on the data types respectively corresponding to the data units.
In a possible implementation manner, the processing unit is specifically configured to:
determining data processing programs corresponding to the data units respectively based on the data types corresponding to the data units respectively;
and processing the data in the plurality of data units according to the data processing program.
In one possible implementation, the data processing includes any one or combination of processing loading and data inspection.
In a possible implementation manner, the apparatus further includes a second generating unit;
the second generating unit is configured to generate, for each data type, a data processing program corresponding to each data type.
In a possible implementation manner, each data type includes a target data type, and the second generating unit is specifically configured to:
determining data operation parameters corresponding to the target data type, wherein the data operation parameters comprise any one or combination of multiple data operation scene parameters, data processing flow parameters and data processing parameters;
and generating a data processing program corresponding to the target data type according to the data operation parameters.
In a possible implementation manner, the apparatus further includes a second obtaining unit and a sending unit:
the second obtaining unit is used for obtaining a data processing result corresponding to the data processing;
and the sending unit is used for sending the data processing result to a responsible person corresponding to the data set to be processed.
In a third aspect, an embodiment of the present application discloses a computer device, including a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the data processing method of any one of the first aspect according to instructions in the program code.
In a fourth aspect, an embodiment of the present application discloses a computer-readable storage medium, which is used for storing a computer program, where the computer program is used for executing the data processing method in any one of the first aspect.
In a fifth aspect, an embodiment of the present application discloses a computer program product including instructions, which when run on a computer, cause the computer to execute the data processing method of any one of the first aspects.
According to the technical scheme, after the data set to be processed is obtained, the processing device may analyze a plurality of data types corresponding to the data included in the data set to be processed, and then generate a plurality of data units corresponding to the data set to be processed, where the data types included in the same data unit are the same. Subsequently, the processing device may perform data processing on data included in the plurality of data units based on the data types respectively corresponding to the plurality of data units, thereby enabling efficient data processing for data including a plurality of data types.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present application;
fig. 2 is a flowchart of a data processing method according to an embodiment of the present application;
fig. 3 is a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
Detailed Description
Embodiments of the present application are described below with reference to the accompanying drawings.
It is understood that the method may be applied to a processing device, which is a processing device capable of data processing, for example, a terminal device or a server having a data processing function. The method can be independently executed through the terminal equipment or the server, can also be applied to a network scene of communication between the terminal equipment and the server, and is executed through the cooperation of the terminal equipment and the server. The terminal device may be a computer, a mobile phone, or the like. The server may be understood as an application server or a Web server, and in actual deployment, the server may be an independent server or a cluster server.
Next, a data processing method provided by an embodiment of the present application will be described with reference to the drawings.
Referring to fig. 1, fig. 1 is a flowchart of a data processing method provided in an embodiment of the present application, where the method includes:
s101: and acquiring a data set to be processed.
Wherein the set of data to be processed includes data corresponding to a plurality of data types.
S102: and generating a plurality of data units corresponding to the data set to be processed.
In order to efficiently process a data set including multiple data types, the processing device may first analyze the data types of data included in the data set to be processed, and perform packet processing on the data in the data set to be processed based on the data types to obtain multiple data units corresponding to the data set to be processed, where the data types of the data included in the same data unit are the same.
S103: and processing data included in the plurality of data units based on the data types respectively corresponding to the plurality of data units.
After the data classification, the processing device can perform data processing on the data of the data units in a targeted manner based on the data types respectively corresponding to the data units, so that efficient data processing can be performed even if multiple types of comprehensive data are obtained.
According to the technical scheme, after the data set to be processed is obtained, the processing device may analyze a plurality of data types corresponding to the data included in the data set to be processed, and then generate a plurality of data units corresponding to the data set to be processed, where the data types included in the same data unit are the same. Subsequently, the processing device may perform data processing on data included in the plurality of data units based on the data types respectively corresponding to the plurality of data units, thereby enabling efficient data processing for data including a plurality of data types.
Specifically, in one possible implementation manner, when performing data processing, the processing device may determine, based on the data types respectively corresponding to the plurality of data units, data processing programs respectively corresponding to the plurality of data units, where the data processing programs are used for performing data processing on data of a certain data type. The processing device may perform data processing on the data in the plurality of data units according to the data processing program, that is, perform targeted data processing through the data processing program corresponding to each data unit.
The content of the data processing may include various types, and in a possible implementation manner, the data processing may include any one or combination of more of processing loading and data checking.
In a possible implementation manner, in order to perform targeted data processing on data of multiple data types, before acquiring a data set to be processed, the processing device may generate a data processing program corresponding to each data type for each data type. Specifically, in a possible implementation manner, each data type may include a target data type, where the target data type may be any one of the data types, and when a data processing program is generated, for the target data type, the processing device may first determine a data operation parameter corresponding to the target data type, where the data operation parameter includes any one or a combination of multiple data operation scene parameters, data processing flow parameters, and data processing parameters. The data operation scene parameter is used for identifying an operation scene corresponding to the data of the data type, the data processing flow parameter is used for identifying a processing flow required for processing the data of the data type, and the data processing parameter is used for identifying a processing mode of the data type. The processing device can generate a data processing program corresponding to the target data type according to the data operation parameters, so that the data processing program can efficiently process the data of the target data type.
In addition, in a possible implementation manner, after data processing is performed, the processing device may further obtain a data processing result corresponding to the data processing, and then send the data processing result to a responsible person corresponding to the set of data to be processed, so that a relevant person can know the data processing situation at the first time.
For example, as shown in fig. 2, fig. 2 is a flowchart of a data processing method provided in an embodiment of the present application, where the method includes:
s201: and entering a data gateway program.
The data gateway program comprises a data processing program aiming at a plurality of data types. For a specific service data set, the processing device may perform parameter configuration such as flow step configuration, processing method, scene customization, etc., to obtain the data processing program, and may perform dynamic processing in different modes for different data units.
S202: initializing and acquiring a data set, and performing logic judgment processing according to parameter configuration.
S203: and analyzing the data set, and splitting the data set into specific data units.
S204: and the data gateway circularly performs the back line calling logic processing of the Nth data unit (N > ═ 1, and 1 is a single data unit).
And analyzing and acquiring different uploaded data units, calling a data processing program in a routing mode and the like, analyzing text data, splitting data, and calling a subsequent module to perform service processing.
S205: and sorting the processing results of the summarized data units.
S206: communicating with peripheral systems.
The data processing mode can be used for realizing different technical means for different system platforms and maintaining different data units.
Based on the data processing method provided in the foregoing embodiment, an embodiment of the present application further provides a data processing apparatus, referring to fig. 3, and fig. 3 is a block diagram of a structure of a data processing apparatus 300 provided in the embodiment of the present application, where the apparatus includes a first obtaining unit 301, a first generating unit 302, and a processing unit 303:
the first obtaining unit 301 is configured to obtain a to-be-processed data set, where the to-be-processed data set includes data corresponding to multiple data types;
the first generating unit 302 is configured to generate a plurality of data units corresponding to the to-be-processed data set, where data types of data included in the same data unit are the same;
the processing unit 303 is configured to perform data processing on data included in the plurality of data units based on data types respectively corresponding to the plurality of data units.
In a possible implementation manner, the processing unit 303 is specifically configured to:
determining data processing programs respectively corresponding to the data units based on the data types respectively corresponding to the data units;
and processing the data in the data units according to the data processing program.
In one possible implementation, the data processing includes any one or combination of processing loading and data inspection.
In a possible implementation manner, the apparatus further includes a second generating unit;
the second generating unit is configured to generate, for each data type, a data processing program corresponding to each data type.
In a possible implementation manner, each data type includes a target data type, and the second generating unit is specifically configured to:
determining data operation parameters corresponding to the target data type, wherein the data operation parameters comprise any one or combination of multiple data operation scene parameters, data processing flow parameters and data processing parameters;
and generating a data processing program corresponding to the target data type according to the data operation parameters.
In a possible implementation manner, the apparatus further includes a second obtaining unit and a sending unit:
the second obtaining unit is used for obtaining a data processing result corresponding to the data processing;
and the sending unit is used for sending the data processing result to a responsible person corresponding to the data set to be processed.
The embodiment of the present application further provides a computer device, which may also be a terminal device, where the terminal device may be any intelligent terminal including a mobile phone, a tablet computer, a Personal Digital Assistant (PDA for short), a Point of Sales (POS for short), a vehicle-mounted computer, and the like,
in this embodiment, the processor included in the terminal device further has the following functions:
acquiring a data set to be processed, wherein the data set to be processed comprises data corresponding to a plurality of data types;
generating a plurality of data units corresponding to the data set to be processed, wherein the data types of the data included in the same data unit are the same;
and processing data included in the data units based on the data types respectively corresponding to the data units.
In addition, the embodiment of the present application further provides a storage medium, where the storage medium is used to store a computer program, and the computer program is used to execute the data processing method provided by the foregoing embodiment.
Embodiments of the present application further provide a computer program product including instructions, which when run on a computer, cause the computer to execute the data processing method provided by the above embodiments.
Those of ordinary skill in the art will understand that: all or part of the steps for realizing the method embodiments can be completed by hardware related to program instructions, the program can be stored in a computer readable storage medium, and the program executes the steps comprising the method embodiments when executed; and the aforementioned storage medium may be at least one of the following media: various media that can store program codes, such as read-only memory (ROM), RAM, magnetic disk, or optical disk.
It should be noted that, in the present specification, all the embodiments are described in a progressive manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus and system embodiments, since they are substantially similar to the method embodiments, they are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described embodiments of the apparatus and system are merely illustrative, and the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only one specific embodiment of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present application should be covered within the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (10)
1. A method of data processing, the method comprising:
acquiring a data set to be processed, wherein the data set to be processed comprises data corresponding to a plurality of data types;
generating a plurality of data units corresponding to the data set to be processed, wherein the data types of the data included in the same data unit are the same;
and processing data included in the data units based on the data types respectively corresponding to the data units.
2. The method according to claim 1, wherein the performing data processing on the data included in the plurality of data units based on the data types respectively corresponding to the plurality of data units comprises:
determining data processing programs respectively corresponding to the data units based on the data types respectively corresponding to the data units;
and processing the data in the data units according to the data processing program.
3. The method of claim 1, wherein the data processing comprises any one or combination of processing loading and data inspection.
4. The method of claim 2, wherein prior to said obtaining a set of data to be processed, the method further comprises;
and generating a data processing program corresponding to each data type aiming at each data type.
5. The method according to claim 4, wherein each data type includes a target data type, and the generating of the data processing program corresponding to each data type includes:
determining data operation parameters corresponding to the target data type, wherein the data operation parameters comprise any one or combination of multiple data operation scene parameters, data processing flow parameters and data processing parameters;
and generating a data processing program corresponding to the target data type according to the data operation parameters.
6. The method of claim 1, further comprising:
acquiring a data processing result corresponding to the data processing;
and sending the data processing result to a responsible person corresponding to the data set to be processed.
7. A data processing apparatus, characterized in that the apparatus comprises a first acquisition unit, a first generation unit, and a processing unit:
the first acquisition unit is used for acquiring a data set to be processed, and the data set to be processed comprises data corresponding to a plurality of data types;
the first generating unit is used for generating a plurality of data units corresponding to the data set to be processed, and the data types of the data included in the same data unit are the same;
the processing unit is configured to perform data processing on data included in the plurality of data units based on data types respectively corresponding to the plurality of data units.
8. A computer device, comprising a processor and a memory:
the memory is used for storing program codes and transmitting the program codes to the processor;
the processor is configured to perform the data processing method of any one of claims 1-6 according to instructions in the program code.
9. A computer-readable storage medium for storing a computer program for executing the data processing method of any one of claims 1 to 6.
10. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the data processing method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210405525.8A CN115017216A (en) | 2022-04-18 | 2022-04-18 | Data processing method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210405525.8A CN115017216A (en) | 2022-04-18 | 2022-04-18 | Data processing method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115017216A true CN115017216A (en) | 2022-09-06 |
Family
ID=83067697
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210405525.8A Pending CN115017216A (en) | 2022-04-18 | 2022-04-18 | Data processing method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115017216A (en) |
-
2022
- 2022-04-18 CN CN202210405525.8A patent/CN115017216A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11232392B2 (en) | Method for processing orders and electronic device | |
| CN107886414B (en) | Order combination method and equipment and computer storage medium | |
| CN107908680A (en) | Management method, electronic device and the computer-readable recording medium of wechat public platform | |
| CN111954173A (en) | Method, device, server and computer readable storage medium for sending short message | |
| CN109857524B (en) | Stream computing method, device, equipment and computer readable storage medium | |
| CN112035344A (en) | Multi-scenario test method, device, equipment and computer readable storage medium | |
| CN110428840A (en) | Implementation of load balancing, device and computer equipment in speech recognition system | |
| CN111274017B (en) | Resource processing method and device, electronic equipment and storage medium | |
| CN110750295B (en) | Information processing method, device, electronic equipment and storage medium | |
| CN102045206B (en) | Alarm pushing method and device and system thereof | |
| CN111988429A (en) | Algorithm scheduling method and system | |
| CN111193631B (en) | Information processing method, system, and computer-readable storage medium | |
| CN115017216A (en) | Data processing method and related device | |
| CN113095515A (en) | Service fault information processing method and device | |
| CN107944798A (en) | Intelligent goods dispatch method and system | |
| CN111813529B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN112101810A (en) | Risk event control method, device and system | |
| CN113691618A (en) | Message notification method, device, message center and storage medium | |
| CN108932283B (en) | Customer information screening method, system, computer device and storage medium | |
| CN108509478B (en) | Splitting and calling method of rule engine file, electronic device and storage medium | |
| CN114841267B (en) | Real-time prediction method, device, electronic equipment and computer program product | |
| CN111562928A (en) | Resource providing method and device and resource downloading method and device | |
| CN105354696A (en) | OA flow processing method, apparatus and system | |
| CN113157911A (en) | Service verification method and device | |
| CN113055419B (en) | Information sending method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |