CN111475532A - Data processing optimization method and device, storage medium and terminal - Google Patents
Data processing optimization method and device, storage medium and terminal Download PDFInfo
- Publication number
- CN111475532A CN111475532A CN202010147752.6A CN202010147752A CN111475532A CN 111475532 A CN111475532 A CN 111475532A CN 202010147752 A CN202010147752 A CN 202010147752A CN 111475532 A CN111475532 A CN 111475532A
- Authority
- CN
- China
- Prior art keywords
- data
- processing
- feature
- different application
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 189
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000005457 optimization Methods 0.000 title claims abstract description 22
- 238000007499 fusion processing Methods 0.000 claims abstract description 48
- 230000004927 fusion Effects 0.000 claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 42
- 230000000694 effects Effects 0.000 claims description 16
- 238000004891 communication Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 claims description 15
- 230000008569 process Effects 0.000 description 8
- 238000012423 maintenance Methods 0.000 description 6
- 238000013499 data model Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000007477 logistic regression Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an optimization method and device for data processing, a storage medium and a terminal, relates to the technical field of data processing, and mainly aims to solve the problems of low efficiency and low accuracy of the existing single application platform on the processing of data with relevance through a model. The method comprises the following steps: acquiring characteristic data of a business object from different application platform databases with incidence relations; respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data; and training a preset data processing model according to the processed feature fusion data and the service information acquired from the databases of the different application platforms, and outputting an optimized data processing model, wherein the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements. The method is mainly used for optimizing data processing.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for optimizing data processing, a storage medium, and a terminal.
Background
With the rapid development of big data processing, for a large amount of data generated by user interactive operations such as business transactions on different application platforms, analysis results are obtained through data processing of a background system. For example, a specific model algorithm is utilized to perform sequencing analysis on the business transaction conditions of online merchants in combination with the transaction data of the user.
At present, the data processing implemented for the data in the application platform only depends on the user data of the application platform itself as the basic data for algorithm model training, so as to obtain the data processing result of the application platform by using the trained model to carry out operation, if data correlation exists between different application platforms, for example, data change of one application platform affects data change of another application platform, after the algorithm model is trained by data in a single application platform, the result determined by the algorithm model is relatively independent, so that a large amount of data is lost, the model prediction in a service scene is inaccurate, the data relevance of the relevant data in different application platforms cannot be reflected, the constructed model algorithm has larger training data aiming at different application platforms and high maintenance cost, thereby influencing the accuracy and efficiency of data processing by using the algorithm model.
Disclosure of Invention
In view of this, the present invention provides an optimization method and apparatus for data processing, a storage medium, and a terminal, and mainly aims to solve the problems of low efficiency and low accuracy of processing data with relevance through a model pair in the existing single application platform.
According to an aspect of the present invention, there is provided a data processing optimization method, including:
acquiring characteristic data of a business object from different application platform databases with incidence relations;
respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data;
and training a preset data processing model according to the processed feature fusion data and the service information acquired from the databases of the different application platforms, and outputting an optimized data processing model, wherein the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
Further, the performing feature fusion processing on the feature data according to the data structure type of the generated feature data includes:
dividing the feature data into data structure types;
and carrying out feature fusion processing on the feature data which belong to different application platforms and have the same data structure type.
Further, the performing of feature fusion processing on the feature data belonging to different application platforms and the same data structure type includes:
determining data structures of the characteristic data belonging to different application platforms and of the same data structure type, and performing data weighting processing and/or data splicing processing on the characteristic data according to the data structures.
Further, before performing feature fusion processing on the feature data belonging to different application platforms and of the same data structure type, the method further includes:
judging whether the characteristic data belonging to different application platforms and having the same data structure type exceed a preset fusion deviation range or not;
and if the deviation exceeds the preset fusion deviation range, adjusting a weighting function for carrying out data weighting processing.
Further, before training a preset data processing model according to the processed feature fusion data and the service information acquired from the different application platform databases and outputting the optimized data processing model, the method further includes:
and performing numerical processing on the feature data of the service object and the service information which are subjected to the feature fusion processing, so that the processed feature fusion data is used for training the preset processing model.
Further, the method further comprises:
and extracting the service information of the service object from the different application platform databases according to the data structure of the feature data after feature fusion processing.
Further, the business object includes at least one data source object that generates characterizing data in a business activity.
Further, the method further comprises:
and when a processing instruction of the service requirement is received, outputting a processing result corresponding to the service requirement according to the service information carried in the processing instruction and the data processing model.
According to another aspect of the present invention, there is provided an optimization apparatus for data processing, including:
the acquisition module is used for acquiring the characteristic data of the business object from different application platform databases with incidence relations;
the processing module is used for respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data;
and the training module is used for training a preset data processing model according to the processed feature fusion data and the service information acquired from the databases of the different application platforms and outputting the optimized data processing model, wherein the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
Further, the processing module comprises:
a dividing unit configured to divide a data structure type for the feature data;
and the processing unit is used for carrying out feature fusion processing on the feature data which belong to different application platforms and have the same data structure type.
Further, the processing unit is specifically configured to determine data structures of the feature data belonging to different application platforms and having the same data structure type, and perform data weighting processing and/or data splicing processing on the feature data according to the data structures.
Further, the processing module further comprises:
the judging unit is used for judging whether the characteristic data which belong to different application platforms and have the same data structure type exceed a preset fusion deviation range or not;
and the adjusting unit is used for adjusting a weighting function for carrying out data weighting processing if the preset fusion deviation range is exceeded.
Further, the processing module is further configured to perform numerical processing on the feature data of the service object and the service information, where feature fusion processing is completed, so that the processed feature fusion data is used to train the preset processing model.
Further, the apparatus further comprises:
and the extraction module is used for extracting the service information of the service object from the different application platform databases according to the data structure of the feature data after feature fusion processing.
Further, the business object includes at least one data source object that generates characterizing data in a business activity.
Further, the apparatus further comprises:
and the output module is used for outputting a processing result corresponding to the service demand according to the service information carried in the processing instruction and the data processing model when the processing instruction of the service demand is received.
According to still another aspect of the present invention, there is provided a storage medium having at least one executable instruction stored therein, the executable instruction causing a processor to perform operations corresponding to the optimization method of data processing as described above.
According to still another aspect of the present invention, there is provided a terminal including: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction enables the processor to execute the operation corresponding to the data processing optimization method.
By the technical scheme, the technical scheme provided by the embodiment of the invention at least has the following advantages:
compared with the prior art that the data processing implemented by aiming at the data in the application platform only depends on the user data of the application platform and is used as the basic data for carrying out algorithm model training, the embodiment of the invention acquires the characteristic data of the business object from different application platforms, fuses the characteristic data according to the data structure type, and utilizes the fused characteristic data and the business information to carry out training optimization on the data model, realizes the embodiment of the data processing relevance among multiple application platforms, improves the accuracy of the business processing by utilizing the data processing model, improves the utilization rate of the data, takes the single data processing model after the characteristic fusion as the training model of the multiple application platforms, reduces the resource consumption of the training data of the application platforms and reduces the maintenance cost, therefore, the accuracy and the efficiency of data processing by using the algorithm model are improved.
The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flow chart of an optimization method for data processing according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data processing optimization method provided by the embodiment of the invention;
FIG. 3 is a flowchart illustrating a model training process after feature data fusion in multiple application platforms according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating an apparatus for optimizing data processing according to an embodiment of the present invention;
FIG. 5 is a block diagram of an apparatus for optimizing data processing according to another embodiment of the present invention;
fig. 6 shows a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
An embodiment of the present invention provides an optimization method for data processing, as shown in fig. 1, the method includes:
101. and acquiring characteristic data of the business object from different application platform databases with the incidence relation.
In the embodiment of the present invention, the different application platforms having the association relationship are at least 2 application platforms having data association, for example, when the user 1 generates service data in the service activity performed in the application platform a, the service data is also generated in the application platform b. In addition, the different application platforms having the association relationship may also be at least 2 application platforms having business associations, for example, business associations exist between application platforms respectively providing a takeout service and an arrival service, or business associations exist between application platforms respectively providing an e-commerce application and an e-payment application, that is, business services provided between application platforms are associated, and the embodiment of the present invention is not particularly limited. The business object includes at least one data source object that generates feature data in a business activity, the data source object may be a user, an online merchant, and the like that perform the business activity in an application platform, the business object may include the user and the online merchant, or may include multiple users, or multiple online merchants, and the like for different business activities, for example, for a business that the user performs ordering on the merchants, the business object may include the user and the online merchants, and the embodiment of the present invention is not particularly limited. When the business object is in business, different feature data may be generated according to different business activities, that is, the feature data may be a representation of the user, or a representation of an online merchant, for example, a time characteristic, an identity characteristic, and the like of the user performing the business activities.
It should be noted that feature data generated when different business objects perform business activities in different application platforms are stored in corresponding business platform databases, and therefore, when obtaining the feature data, feature data of different business objects need to be obtained from different application platform databases, for example, feature data of a user a and a merchant a are obtained from databases of the application platform 1, and feature data of the user a and the merchant a are obtained from databases of the application platform 2, respectively.
102. And respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data.
The business object stores the generated characteristic data in the corresponding application platform databases when performing business activities in different application platforms, so that different data structure types can be generated for different characteristic data in the storage process, and in order to enable the characteristic data serving as basic data to serve as a training data optimization model of a data processing model, the characteristic data is subjected to characteristic fusion processing according to the data structure types. The data structure type may include a numerical data structure, a list data structure, a dictionary data structure, and the like, and feature fusion processing may be performed on different types of data structures according to different manners, for example, feature data of the numerical data structure may be directly fused by a weighted average function, which is not limited in the embodiment of the present invention.
It should be noted that, because the business object includes at least one data source object that generates the feature data in the business activity, if the business object is 1 data source object, the feature data of the data source object in the different application platform databases is fused, and if the business object is multiple data source objects, the feature data corresponding to the data source object in the different application platform databases is fused, for example, if the business object includes a user and an online merchant, the feature data of the user in the different application platform databases is fused, and the feature data of the online merchant in the different application platform databases is fused, which is not limited in the embodiment of the present invention.
103. And training a preset data processing model according to the processed feature fusion data and the service information acquired from the different application platform databases, and outputting the optimized data processing model.
In the embodiment of the invention, in order to optimize the data processing model for data processing so as to obtain the data processing model suitable for multiple application platforms, the preset data processing model is trained according to the processed feature fusion data and the service information acquired from different application platform databases. The preset data processing model is a data processing model corresponding to the different application platforms selected according to the service requirements, that is, the different application platforms can use a common data processing model to perform data operation, so that the data processing between the different application platforms is associated, the service requirements include requirements of users on services in different service activities, for example, order service ordering, and the like. In addition, the acquisition of the service information includes all information generated by the service object in the service activity, such as order information, collection information, and the like, and the embodiment of the present invention is not particularly limited.
It should be noted that the preset data processing model in the embodiment of the present invention may be a data processing module corresponding to different application platforms, for example, the application platform 1 and the application platform 2, which are selected according to business requirements ordered by an order, and a machine learning model, such as a logistic regression model, an xgboost model, or the like, which is compared with the application platform 1 and the application platform 2 is selected according to the business requirements ordered by the order, and the model is trained to complete optimization of data processing, so as to obtain an optimized data processing model.
Compared with the prior art that the data processing implemented by aiming at the data in the application platform only depends on the user data of the application platform and is used as the basic data for carrying out algorithm model training, the embodiment of the invention obtains the characteristic data of the business object from different application platforms, feature data are fused according to the data structure type, the data model is trained and optimized by utilizing the fused feature data and the service information, the data processing relevance between multiple application platforms is reflected, the accuracy of service processing by utilizing the data processing model is improved, the utilization rate of the data is improved, the single data processing model after feature fusion is used as the training model of the plurality of application platforms, so that the resource consumption of the training data of the application platforms is reduced, the maintenance cost is reduced, and the accuracy and the efficiency of data processing by using the algorithm model are improved.
An embodiment of the present invention provides another data processing optimization method, as shown in fig. 2, the method includes:
201. and acquiring characteristic data of the business object from different application platform databases with the incidence relation.
This step is the same as step 101 shown in fig. 1, and is not described herein again.
202. Partitioning the feature data into data structure types.
For the embodiment of the present invention, since the data structure types of the feature data stored in the data of different application platforms are different, in order to enable feature data of the same data structure type to be subjected to feature fusion processing, the data structure types need to be divided for the obtained feature data. The data structure type may include a numerical data structure, a list data structure, a dictionary data structure, and the like, and feature fusion processing may be performed on different types of data structures according to different manners, which is not specifically limited in the embodiment of the present invention.
It should be noted that the division of the data structure type may be determined according to storage identifiers stored in different application platform databases for the feature data, for example, if the storage identifier in the database is int, the data structure may be determined as a numeric data structure, if the storage identifier in the database is list, the data structure may be determined as a list-type data structure, and if the storage identifier in the database is dit, the data structure may be determined as a word-type data structure, so that fusion processing in different manners may be performed according to different data structures.
203. And carrying out feature fusion processing on the feature data which belong to different application platforms and have the same data structure type.
For the embodiment of the present invention, in order to combine data having relevance in different application platforms, feature fusion processing needs to be performed on feature data belonging to different application platforms and having the same data structure type. The feature data of different application platforms and the same data structure type are feature data of the same data structure type generated by the business object in the business activity process in different application platforms, for example, if the feature data of the same data structure type generated by the user a in the application platform 1 and the application platform 2 are numerical types 20 and 10, respectively, feature fusion processing is performed on the numerical type feature data 20 and 10.
For further explanation and limitation, in the embodiment of the present invention, to implement feature fusion processing in different ways on feature data of different data structures, step 203 may specifically include: determining data structures of the characteristic data belonging to different application platforms and of the same data structure type, and performing data weighting processing and/or data splicing processing on the characteristic data according to the data structures.
When the feature data of different data structure types are used as a training data set to train a model, the training modes of the data processing model are different, and therefore, the data structures of different types need to be processed according to the corresponding fusion mode. Further, since the data structure type includes a numerical data structure, a list data structure, a dictionary data structure, and the like, when the feature fusion processing is performed, since the numerical feature data includes only numerical values, when the feature fusion processing is performed, the data weighting processing may be performed on the numerical feature data using a weighting function, for example, the user a has a passenger order price of 40 and 38 on the platform 1 and the platform 2, respectively, and then calls the weighted average function g (40, 38) for both numbers. Since the list-type data structure includes numerical values and corresponding storage locations, when performing the feature fusion process, the data fusion process may be performed on feature data belonging to a list type, for example, a list feature1 (id 1, id 2) belonging to the platform 1 of a business preferred by a user, a list failure 2 (id 1, id 4) belonging to the platform 2, and the data fusion process may be performed, so that a feature _ merge after the fusion is [ id1, id2, id4 ]. Since the dictionary-type data structure includes key words and corresponding numerical values, when performing the feature fusion process, the data weighting process and the data concatenation process may be performed on feature data belonging to typical words, for example, when feature1 in platform 1 and feature2 in platform 2 have the same key, the weighting process is performed, when feature1 in platform 1 and feature2 in platform 2 have different keys, the concatenation process is performed, and if feature1 { k1: v1, k2: v2}, feature2 ═ k1: v3, k4: v4}, and the weighted averaging function g (x1, x2) is used for weighted averaging, then feature _ merge ═ k1: g (v1, v3), k4: v 4.
Further, in order to make the feature fusion more accurate and improve the feature fusion efficiency, before step 203, the embodiment of the present invention further includes: judging whether the characteristic data belonging to different application platforms and having the same data structure type exceed a preset fusion deviation range or not; and if the deviation exceeds the preset fusion deviation range, adjusting a weighting function for carrying out data weighting processing.
For example, when a numerical value is fused by using a weighting function, if one of the numerical values is larger and the other numerical value is smaller, the fused numerical value may have a deviation, for example, the exposure of a merchant on the stage 1 is 10, the exposure of the merchant on the stage 2 is 1000, and after the fusion according to the weighting function g (x1, x2) — (x1+ x2)/2, the feature may have a larger deviation for the merchant with a large exposure, so that whether to adjust the weighting function is determined according to the preset fusion deviation range configured. In this embodiment of the present invention, the weighting function is a preset processing function for performing weighting processing, and may be adjusted to g (x1, x2) ═ w1 × 1+ w2 × 2)/(w1+ w2, where w1 and w2 are adjusted weighting coefficients, respectively, which is not limited in this embodiment of the present invention. By adjusting the weighting function, the fusion accuracy of feature data fusion is improved, and the data processing efficiency is improved.
204. And extracting the service information of the service object from the different application platform databases according to the data structure of the feature data after feature fusion processing.
In the embodiment of the invention, in order to match the service information corresponding to the service object acquired from different application platforms with the feature data subjected to the feature fusion processing, the service information of the service object is extracted according to the data structure of the feature data subjected to the feature fusion processing. For example, if the data structure of the feature data after feature fusion is a numerical type, the order information of the user is extracted according to the numerical type, and the embodiment of the present invention is not particularly limited.
205. And carrying out numerical processing on the feature data of the service object and the service information which are subjected to the feature fusion processing.
In the embodiment of the present invention, for feature data subjected to feature fusion processing, in order to train a preset data processing model by using the feature data subjected to feature fusion processing, the feature data needs to be subjected to digitization processing, so that the processed feature fusion data is used for training the preset processing model. The feature data and the service information may be merged, and the feature data after feature fusion is added to the service object as a primary key, so as to perform a unified digitization process, for example, the user id and the merchant id are primary keys, and the feature data after feature fusion is merged to obtain [ id, merge _ feature 1.
In the embodiment of the present invention, the numeralization may be calculated by a preset numeralization formula, for example, the preset numeralization formula is:
xpercentil(98)is 98 quantile, xminIs the minimum value of x.
206. And training a preset data processing model according to the processed feature fusion data and the service information acquired from the different application platform databases, and outputting the optimized data processing model.
This step is the same as step 103 shown in fig. 1, and is not described herein again. For example, fig. 3 shows a flowchart of model training after feature data fusion in multiple application platforms.
207. And when a processing instruction of the service requirement is received, outputting a processing result corresponding to the service requirement according to the service information carried in the processing instruction and the data processing model.
In the embodiment of the invention, in order to realize optimization of data processing and improve the accuracy and efficiency of data processing under different service requirements, when a processing instruction of the service requirement is received, a processing result is output according to the carried service information and the data processing model. For example, when a processing instruction of ordering of merchants in 2 application platforms is received, data corresponding to the merchants and users are called according to order information carried in the instruction, and processing results suitable for the 2 application platforms are output by using a data processing model for calculating the ordering of the merchants.
Compared with the prior art that the data processing implemented by aiming at the data in the application platform only depends on the user data of the application platform and is used as the basic data for carrying out algorithm model training, the embodiment of the invention obtains the characteristic data of the business object from different application platforms, feature data are fused according to the data structure type, the data model is trained and optimized by utilizing the fused feature data and the service information, the data processing relevance between multiple application platforms is reflected, the accuracy of service processing by utilizing the data processing model is improved, the utilization rate of the data is improved, the single data processing model after feature fusion is used as the training model of the plurality of application platforms, so that the resource consumption of the training data of the application platforms is reduced, the maintenance cost is reduced, and the accuracy and the efficiency of data processing by using the algorithm model are improved.
Further, as an implementation of the method shown in fig. 1, an embodiment of the present invention provides an optimization apparatus for data processing, as shown in fig. 4, the apparatus includes: an acquisition module 31, a processing module 32, and a training module 33.
An obtaining module 31, configured to obtain feature data of a business object from different application platform databases having an association relationship;
the processing module 32 is configured to perform feature fusion processing on the feature data according to the data structure type of the generated feature data;
and the training module 33 is configured to train a preset data processing model according to the processed feature fusion data and the service information obtained from the databases of the different application platforms, and output an optimized data processing model, where the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
Compared with the prior art that the data processing implemented by aiming at the data in the application platform only depends on the user data of the application platform and is used as the basic data for carrying out algorithm model training, the embodiment of the invention obtains the characteristic data of the business object from different application platforms, feature data are fused according to the data structure type, the data model is trained and optimized by utilizing the fused feature data and the service information, the data processing relevance between multiple application platforms is reflected, the accuracy of service processing by utilizing the data processing model is improved, the utilization rate of the data is improved, the single data processing model after feature fusion is used as the training model of the plurality of application platforms, so that the resource consumption of the training data of the application platforms is reduced, the maintenance cost is reduced, and the accuracy and the efficiency of data processing by using the algorithm model are improved.
Further, as an implementation of the method shown in fig. 2, another data processing optimization apparatus is provided in an embodiment of the present invention, as shown in fig. 5, the apparatus includes: an acquisition module 41, a processing module 42, a training module 43, an extraction module 44, and an output module 45.
An obtaining module 41, configured to obtain feature data of a business object from different application platform databases having an association relationship;
the processing module 42 is configured to perform feature fusion processing on the feature data according to the data structure type of the generated feature data;
and a training module 43, configured to train a preset data processing model according to the processed feature fusion data and the service information obtained from the databases of the different application platforms, and output an optimized data processing model, where the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
Further, the processing module 42 includes:
a dividing unit 4201, configured to divide the feature data into data structure types;
the processing unit 4202 is configured to perform feature fusion processing on the feature data belonging to different application platforms and having the same data structure type.
Further, the processing unit 4202 is specifically configured to determine data structures of the feature data belonging to different application platforms and having the same data structure type, and perform data weighting processing and/or data splicing processing on the feature data according to the data structures.
Further, the processing module 42 further includes:
a determining unit 4203, configured to determine whether the feature data belonging to different application platforms and having the same data structure type exceeds a preset fusion deviation range;
an adjusting unit 4204, configured to adjust a weighting function for performing data weighting processing if the predetermined fusion deviation range is exceeded.
Further, the processing module 42 is further configured to perform numerical processing on the feature data of the service object and the service information that have been subjected to the feature fusion processing, so that the processed feature fusion data is used to train the preset processing model.
Further, the apparatus further comprises:
and the extraction module 44 is configured to extract the service information of the service object from the different application platform databases according to the data structure of the feature data after the feature fusion processing is performed.
Further, the business object includes at least one data source object that generates characterizing data in a business activity.
Further, the apparatus further comprises:
and the output module 45 is configured to, when receiving the processing instruction of the service requirement, output a processing result corresponding to the service requirement according to the service information carried in the processing instruction and the data processing model.
Compared with the prior art that the data processing implemented by aiming at the data in the application platform only depends on the user data of the application platform and is used as the basic data for carrying out algorithm model training, the embodiment of the invention obtains the characteristic data of the business object from different application platforms, feature data are fused according to the data structure type, the data model is trained and optimized by utilizing the fused feature data and the service information, the data processing relevance between multiple application platforms is reflected, the accuracy of service processing by utilizing the data processing model is improved, the utilization rate of the data is improved, the single data processing model after feature fusion is used as the training model of the plurality of application platforms, so that the resource consumption of the training data of the application platforms is reduced, the maintenance cost is reduced, and the accuracy and the efficiency of data processing by using the algorithm model are improved.
According to an embodiment of the present invention, a storage medium is provided, the storage medium storing at least one executable instruction, and the computer executable instruction can execute the optimization method of data processing in any of the above method embodiments.
Fig. 6 is a schematic structural diagram of a terminal according to an embodiment of the present invention, and the specific embodiment of the present invention does not limit the specific implementation of the terminal.
As shown in fig. 6, the terminal may include: a processor (processor)502, a communication interface 504, a memory 506, and a communication bus 508.
Wherein: the processor 502, communication interface 504, and memory 506 communicate with one another via a communication bus 508.
A communication interface 504 for communicating with network elements of other devices, such as clients or other servers.
The processor 502 is configured to execute the program 510, and may specifically perform relevant steps in the above-described data processing optimization method embodiment.
In particular, program 510 may include program code that includes computer operating instructions.
The processor 502 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the invention. The terminal comprises one or more processors, which can be the same type of processor, such as one or more CPUs; or may be different types of processors such as one or more CPUs and one or more ASICs.
And a memory 506 for storing a program 510. The memory 506 may comprise high-speed RAM memory, and may also include non-volatile memory (non-volatile memory), such as at least one disk memory.
The program 510 may specifically be used to cause the processor 502 to perform the following operations:
acquiring characteristic data of a business object from different application platform databases with incidence relations;
respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data;
and training a preset data processing model according to the processed feature fusion data and the service information acquired from the databases of the different application platforms, and outputting an optimized data processing model, wherein the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for optimizing data processing, comprising:
acquiring characteristic data of a business object from different application platform databases with incidence relations;
respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data;
and training a preset data processing model according to the processed feature fusion data and the service information acquired from the databases of the different application platforms, and outputting an optimized data processing model, wherein the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
2. The method according to claim 1, wherein the performing feature fusion processing on the feature data respectively according to the data structure type of the generated feature data comprises:
dividing the feature data into data structure types;
and carrying out feature fusion processing on the feature data which belong to different application platforms and have the same data structure type.
3. The method according to claim 2, wherein the performing feature fusion processing on the feature data belonging to different application platforms and the same data structure type comprises:
determining data structures of the characteristic data belonging to different application platforms and of the same data structure type, and performing data weighting processing and/or data splicing processing on the characteristic data according to the data structures.
4. The method according to claim 2, wherein before performing the feature fusion processing on the feature data belonging to different application platforms and the same data structure type, the method further comprises:
judging whether the characteristic data belonging to different application platforms and having the same data structure type exceed a preset fusion deviation range or not;
and if the deviation exceeds the preset fusion deviation range, adjusting a weighting function for carrying out data weighting processing.
5. The method according to claim 1, wherein before training a preset data processing model according to the processed feature fusion data and the service information obtained from the databases of different application platforms and outputting the optimized data processing model, the method further comprises:
and performing numerical processing on the feature data of the service object and the service information which are subjected to the feature fusion processing, so that the processed feature fusion data is used for training the preset processing model.
6. The method of claim 1, further comprising:
and extracting the service information of the service object from the different application platform databases according to the data structure of the feature data after feature fusion processing.
7. The method of any of claims 1-6, wherein the business object comprises at least one data source object that generates characterizing data in a business activity.
8. An apparatus for optimizing data processing, comprising:
the acquisition module is used for acquiring the characteristic data of the business object from different application platform databases with incidence relations;
the processing module is used for respectively carrying out feature fusion processing on the feature data according to the data structure type of the generated feature data;
and the training module is used for training a preset data processing model according to the processed feature fusion data and the service information acquired from the databases of the different application platforms and outputting the optimized data processing model, wherein the preset data processing model is a data processing model corresponding to the different application platforms selected according to service requirements.
9. A storage medium having stored therein at least one executable instruction for causing a processor to perform operations corresponding to the optimization method of data processing according to any one of claims 1 to 7.
10. A terminal, comprising: the system comprises a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface complete mutual communication through the communication bus;
the memory is used for storing at least one executable instruction, and the executable instruction causes the processor to execute the operation corresponding to the optimization method of the data processing according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010147752.6A CN111475532B (en) | 2020-03-05 | 2020-03-05 | Data processing optimization method and device, storage medium and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010147752.6A CN111475532B (en) | 2020-03-05 | 2020-03-05 | Data processing optimization method and device, storage medium and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111475532A true CN111475532A (en) | 2020-07-31 |
| CN111475532B CN111475532B (en) | 2023-11-03 |
Family
ID=71747292
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010147752.6A Active CN111475532B (en) | 2020-03-05 | 2020-03-05 | Data processing optimization method and device, storage medium and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111475532B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609096A (en) * | 2021-07-19 | 2021-11-05 | 北京智思迪科技有限公司 | Data processing method and device |
| CN113626415A (en) * | 2021-08-27 | 2021-11-09 | 天元大数据信用管理有限公司 | Credit data output method, device and medium |
| CN115129992A (en) * | 2022-07-04 | 2022-09-30 | 北京深演智能科技股份有限公司 | Data processing method and device, nonvolatile storage medium and processor |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060204107A1 (en) * | 2005-03-04 | 2006-09-14 | Lockheed Martin Corporation | Object recognition system using dynamic length genetic training |
| CN101901252A (en) * | 2010-06-23 | 2010-12-01 | 中兴通讯股份有限公司 | Method for integrating same user data on multiple websites and integration platform |
| CN106407416A (en) * | 2016-09-23 | 2017-02-15 | 维沃移动通信有限公司 | An application program data integration method and a mobile terminal |
| CN106980999A (en) * | 2016-01-19 | 2017-07-25 | 阿里巴巴集团控股有限公司 | The method and apparatus that a kind of user recommends |
| CN107464174A (en) * | 2017-08-21 | 2017-12-12 | 合肥工业大学 | A kind of take-away platform based on commercial circle, take out meal ordering system and method for ordering |
| CN108510421A (en) * | 2018-01-29 | 2018-09-07 | 湖北省楚天云有限公司 | A kind of government affairs service system, method, equipment and medium |
| CN108959369A (en) * | 2018-05-22 | 2018-12-07 | 国网浙江省电力有限公司电力科学研究院 | A kind of mass data platform and big data platform fusion method |
| CN108984733A (en) * | 2018-07-13 | 2018-12-11 | 北京京东金融科技控股有限公司 | cross-domain data fusion method, system and storage medium |
| CN109146606A (en) * | 2018-07-09 | 2019-01-04 | 广州品唯软件有限公司 | A kind of brand recommended method, electronic equipment, storage medium and system |
| CN109493319A (en) * | 2018-10-10 | 2019-03-19 | 武汉联影医疗科技有限公司 | Blending image effect quantization method, device, computer equipment and storage medium |
| CN109726321A (en) * | 2019-01-11 | 2019-05-07 | 安徽爱吉泰克科技有限公司 | It is a kind of to visualize integrated cloud computing application system and method |
| CN110069988A (en) * | 2019-01-31 | 2019-07-30 | 中国平安财产保险股份有限公司 | AI based on multidimensional data drives risk analysis method, server and storage medium |
| CN110119413A (en) * | 2019-04-30 | 2019-08-13 | 京东城市(南京)科技有限公司 | The method and apparatus of data fusion |
| CN110532254A (en) * | 2018-05-25 | 2019-12-03 | 杭州海康威视数字技术股份有限公司 | The method and apparatus of fused data table |
| CN110704543A (en) * | 2019-08-19 | 2020-01-17 | 上海机电工程研究所 | Multi-type multi-platform information data self-adaptive fusion system and method |
-
2020
- 2020-03-05 CN CN202010147752.6A patent/CN111475532B/en active Active
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060204107A1 (en) * | 2005-03-04 | 2006-09-14 | Lockheed Martin Corporation | Object recognition system using dynamic length genetic training |
| CN101901252A (en) * | 2010-06-23 | 2010-12-01 | 中兴通讯股份有限公司 | Method for integrating same user data on multiple websites and integration platform |
| CN106980999A (en) * | 2016-01-19 | 2017-07-25 | 阿里巴巴集团控股有限公司 | The method and apparatus that a kind of user recommends |
| CN106407416A (en) * | 2016-09-23 | 2017-02-15 | 维沃移动通信有限公司 | An application program data integration method and a mobile terminal |
| CN107464174A (en) * | 2017-08-21 | 2017-12-12 | 合肥工业大学 | A kind of take-away platform based on commercial circle, take out meal ordering system and method for ordering |
| CN108510421A (en) * | 2018-01-29 | 2018-09-07 | 湖北省楚天云有限公司 | A kind of government affairs service system, method, equipment and medium |
| CN108959369A (en) * | 2018-05-22 | 2018-12-07 | 国网浙江省电力有限公司电力科学研究院 | A kind of mass data platform and big data platform fusion method |
| CN110532254A (en) * | 2018-05-25 | 2019-12-03 | 杭州海康威视数字技术股份有限公司 | The method and apparatus of fused data table |
| CN109146606A (en) * | 2018-07-09 | 2019-01-04 | 广州品唯软件有限公司 | A kind of brand recommended method, electronic equipment, storage medium and system |
| CN108984733A (en) * | 2018-07-13 | 2018-12-11 | 北京京东金融科技控股有限公司 | cross-domain data fusion method, system and storage medium |
| CN109493319A (en) * | 2018-10-10 | 2019-03-19 | 武汉联影医疗科技有限公司 | Blending image effect quantization method, device, computer equipment and storage medium |
| CN109726321A (en) * | 2019-01-11 | 2019-05-07 | 安徽爱吉泰克科技有限公司 | It is a kind of to visualize integrated cloud computing application system and method |
| CN110069988A (en) * | 2019-01-31 | 2019-07-30 | 中国平安财产保险股份有限公司 | AI based on multidimensional data drives risk analysis method, server and storage medium |
| CN110119413A (en) * | 2019-04-30 | 2019-08-13 | 京东城市(南京)科技有限公司 | The method and apparatus of data fusion |
| CN110704543A (en) * | 2019-08-19 | 2020-01-17 | 上海机电工程研究所 | Multi-type multi-platform information data self-adaptive fusion system and method |
Non-Patent Citations (1)
| Title |
|---|
| 李莉等: "自适应加权数据融合加权因子的动态调整", no. 3, pages 109 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113609096A (en) * | 2021-07-19 | 2021-11-05 | 北京智思迪科技有限公司 | Data processing method and device |
| CN113626415A (en) * | 2021-08-27 | 2021-11-09 | 天元大数据信用管理有限公司 | Credit data output method, device and medium |
| CN113626415B (en) * | 2021-08-27 | 2024-02-23 | 天元大数据信用管理有限公司 | Credit data output method, equipment and medium |
| CN115129992A (en) * | 2022-07-04 | 2022-09-30 | 北京深演智能科技股份有限公司 | Data processing method and device, nonvolatile storage medium and processor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111475532B (en) | 2023-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111475532B (en) | Data processing optimization method and device, storage medium and terminal | |
| CN111371858B (en) | Group control equipment identification method, device, medium and electronic equipment | |
| CN108388512B (en) | Data processing method and server | |
| WO2022041834A1 (en) | Transaction data processing method and apparatus | |
| CN114896291A (en) | Training method and sequencing method of multi-agent model | |
| CN116955538B (en) | Medical dictionary data matching method and device, electronic equipment and storage medium | |
| CN112148880A (en) | Customer service dialogue corpus clustering method, system, equipment and storage medium | |
| CN114185938B (en) | Project traceability analysis method and system based on digital finance and big data traceability | |
| CN113657817B (en) | Transaction processing method and device, electronic equipment and readable storage medium | |
| CN110175182B (en) | Data checking method and device | |
| CN114840183A (en) | Micro front end adjusting method and device based on user behaviors | |
| CN113379499A (en) | Article screening method and apparatus, electronic device, and storage medium | |
| CN117150215B (en) | Assessment result determining method and device, electronic equipment and storage medium | |
| CN110728125B (en) | Leave-asking form generation method, device, server and storage medium | |
| KR102449831B1 (en) | Electronic device for providing information regarding new text, server for identifying new text and operation method thereof | |
| CN107844521B (en) | Method and device for inquiring transaction result | |
| CN108614871B (en) | Data query and result generation method, device, equipment and storage medium | |
| CN116703505A (en) | Order information judging method and device | |
| CN111858832A (en) | Dialogue method, dialogue device, electronic equipment and storage medium | |
| CN117851390A (en) | Blank certificate processing method, device, equipment and storage medium | |
| CN112269860A (en) | Automatic response processing method and device, electronic equipment and readable storage medium | |
| CN117807056A (en) | Data auditing method and device, electronic equipment and storage medium | |
| CN116319716A (en) | Information processing method, no-service system, electronic device, and storage medium | |
| CN111967960A (en) | Batch adjustment method and device for account data | |
| CN117633088A (en) | File data importing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |