CN117236801B - Data processing method, device, electronic equipment and readable storage medium - Google Patents

Data processing method, device, electronic equipment and readable storage medium Download PDF

Info

Publication number
CN117236801B
CN117236801B CN202311509838.9A CN202311509838A CN117236801B CN 117236801 B CN117236801 B CN 117236801B CN 202311509838 A CN202311509838 A CN 202311509838A CN 117236801 B CN117236801 B CN 117236801B
Authority
CN
China
Prior art keywords
data
target
project
quantitative
equation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311509838.9A
Other languages
Chinese (zh)
Other versions
CN117236801A (en
Inventor
何瑞霞
白雪峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Shunshi Sicheng Technology Co ltd
Original Assignee
Beijing Shunshi Sicheng Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Shunshi Sicheng Technology Co ltd filed Critical Beijing Shunshi Sicheng Technology Co ltd
Priority to CN202311509838.9A priority Critical patent/CN117236801B/en
Publication of CN117236801A publication Critical patent/CN117236801A/en
Application granted granted Critical
Publication of CN117236801B publication Critical patent/CN117236801B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Abstract

The embodiment of the application provides a data processing method, a data processing device, electronic equipment and a readable storage medium, and relates to the field of data processing. Obtaining a plurality of qualitative data, and quantizing the qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data; dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level; acquiring project data requirements, and grading and distributing the initial data volume by using a preset data grading model according to the project data requirements to obtain target data volumes corresponding to the quantization levels; and determining data to be uploaded according to the target data quantity, and uploading the data to be uploaded to a project. The method can make the accuracy of score calculation higher, and improve the utilization rate of the project to the data, thereby reducing the cost.

Description

Data processing method, device, electronic equipment and readable storage medium
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a data processing method, a data processing device, an electronic device, and a readable storage medium.
Background
The data uploading system performs qualitative judgment according to the scoring index of the data, for example: when the data A enters the database, the system can judge according to the set scoring index, and automatically calculate a total scoring result. However, the quality of each piece of data cannot be objectively and accurately reflected by the scoring index, and when the data is uploaded, the uploaded data is controlled according to artificial experience, so that the utilization rate of the data by the project is low, and the cost is high.
Disclosure of Invention
The application provides a data processing method, a data processing device, electronic equipment and a readable storage medium, which can enable the accuracy of score calculation to be high, improve the utilization rate of items to data and reduce the cost.
The technical scheme of the embodiment of the application is as follows:
in a first aspect, an embodiment of the present application provides a data processing method, where the method includes:
obtaining a plurality of qualitative data, and quantizing the qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data;
dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level;
Acquiring project data requirements, and grading and distributing the initial data volume by using a preset data grading model according to the project data requirements to obtain target data volumes corresponding to the quantization levels;
and determining data to be uploaded according to the target data quantity, and uploading the data to be uploaded to a project.
In the technical scheme, a plurality of qualitative data are obtained, the qualitative data are quantized to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data, each piece of data can be assigned through quantization, the accuracy of data scoring is improved, the importance of the data is determined through the quantization levels, and a basis is provided for intelligent uploading of the data; dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level, so that the data can cover each quantization level, and the utilization rate of the data is improved; acquiring project data requirements, grading and distributing initial data volume by using a preset data grading model according to the project data requirements to obtain target data volume corresponding to each quantization level, and performing data distribution by using a data grading model according to the project requirements so that the obtained target data volume can uniformly cover the data of each quantization level, thereby improving the data utilization rate and reducing the cost; and determining data to be uploaded according to each target data amount, and uploading the data to be uploaded to the project, so that automatic matching uploading can be realized.
In some embodiments of the present application, the quantifying the qualitative data to obtain a plurality of quantitative data, and a quantization level corresponding to each of the quantitative data includes:
acquiring a preset scoring index;
configuring weights for the scoring indexes by using a preset weight assignment algorithm to obtain weight ratios;
scoring the qualitative data according to the weight ratio to obtain quantitative data, wherein the quantitative data is characterized as scoring values;
and obtaining the quantization grade based on a preset grade score section and the quantitative data.
In the technical scheme, the weight assignment algorithm is utilized to configure weights for the scoring indexes, and different scoring indexes are concerned differently, so that subsequent scoring is facilitated, and scoring accuracy is improved. Scoring is carried out according to the weight ratio, quantitative data are obtained, and a quantization grade is obtained based on a preset grade score section and the quantitative data, so that the subsequent improvement of the data utilization rate is facilitated.
In some embodiments of the present application, the weight assignment algorithm is one of a principal component analysis method, an AHP hierarchy analysis method, and a combined weighting method.
In the above technical solution, the main data or the common data can be focused by one of the principal component analysis method, the AHP hierarchical analysis method and the combined weighting method, so that the degree of importance of different indexes can be reflected.
In some embodiments of the present application, in a case where the project data requirement is a data requirement corresponding to one target project, the project data requirement includes a first target upload amount and a first target effective rate;
and grading and distributing the initial data volume by using a preset data grading model according to the project data requirement to obtain target data volumes corresponding to the quantization levels, wherein the grading and distributing comprises the following steps:
carrying out equation construction on the initial data quantity, the first target uploading quantity, the first target effective rate and the quantization level by using a preset data scoring model to obtain a first equation;
and solving according to the first equation to obtain a solution of an equation, and optimizing the solution of the equation to obtain the target data quantity corresponding to each quantization level.
According to the technical scheme, according to project data requirements, an equation is built by using a preset data scoring model, equation solving and optimizing are performed, and data distribution is automatically performed through the data scoring model, so that the data to be uploaded can be determined later, and the data utilization rate is improved.
In some embodiments of the present application, the quantization levels include a level, B level, C level, D level, E level;
The equation construction is performed on the initial data volume, the first target uploading volume, the first target effective rate and the quantization level by using a preset data scoring model to obtain a first equation, including:
the first mode is as follows:
wherein N is expressed as the first target uploading amount, r is expressed as the first target effective rate,each represented as said initial data amount corresponding to each of said quantization levels,and the weight ratio corresponding to each quantization level is shown.
In the above technical scheme, the distribution of the data can be clearly reflected by the constructed equation, and the quality of the data can be reflected.
In some embodiments of the present application, in a case where the project data requirement is a data requirement corresponding to a plurality of target projects, the project data requirement includes a second target upload amount and a second target effective rate corresponding to each of the target projects;
and grading and distributing the initial data volume by using a preset data grading model according to the project data requirement to obtain target data volumes corresponding to the quantization levels, wherein the grading and distributing comprises the following steps:
carrying out equation construction on the initial data volume, the second target uploading volume, the second target effective rate and the quantization level by using a preset data scoring model to obtain a second equation and a conditional constraint inequality, wherein the conditional constraint inequality is obtained by determining the initial data volume corresponding to the quantization level and the second target uploading volume corresponding to each target item;
And obtaining a solution set corresponding to the target item according to the second equation and the condition constraint inequality, and calculating an optimal ratio based on the solution set to obtain the target data volume.
In the above technical scheme, under the condition that the project data requirement is the data requirement corresponding to a plurality of target projects, a second equation and a condition constraint inequality are constructed by using a preset data scoring model according to the project data requirement, and a solution set is obtained according to the second equation and the condition constraint inequality, so that the data uploading processing of the plurality of target projects is facilitated, and then the optimal proportion is calculated based on the solution set, so that the target data quantity is obtained, and the data utilization rate is improved through optimization.
In some embodiments of the present application, the calculating the optimal matching based on the solution set to obtain the target data amount includes:
according to the solution numerical value in the solution set, calculating to obtain a target optimization function, wherein the target optimization function comprises data quality and data utilization rate;
and optimizing the solution set by using the target optimization function to obtain the target data volume.
In the technical scheme, the target optimization function is calculated according to the solution numerical value in the solution set, the target optimization function comprises the data quality and the data utilization rate, the data quality can reflect the satisfaction degree of the user on the data, the target optimization function is utilized for optimization processing, the target data quantity is obtained, and the data utilization rate is improved.
In a second aspect, embodiments of the present application provide a data processing apparatus, the apparatus including:
the data acquisition module is used for acquiring a plurality of qualitative data, quantizing the qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data;
the data dividing module is used for dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level;
the scoring module is used for acquiring project data requirements, and scoring and distributing the initial data volume by using a preset data scoring model according to the project data requirements to obtain target data volumes corresponding to the quantization levels;
and the data uploading module is used for determining data to be uploaded according to the target data quantity and uploading the data to be uploaded to the project.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, a user interface, and a network interface, where the memory is configured to store instructions, and the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory, so that the electronic device performs the method provided in any one of the first aspect above.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium storing instructions that, when executed, perform the method of any one of the first aspects provided above.
In summary, one or more technical solutions provided in the embodiments of the present application at least have the following technical effects or advantages:
1. the quantitative data and the quantitative grades are obtained by quantizing the qualitative data, the quantitative data corresponding to each quantitative grade are scored and distributed according to the project data requirement after the quantitative data and the quantitative grades are divided according to the quantitative grades, the target data quantity is obtained, and the technical means of uploading the data according to the target data quantity are adopted, so that the problem of low project data utilization rate and high cost in the related technology is effectively solved. According to the method and the device for calculating the score value, the accuracy of the score value calculation is high, uploaded data cover different grades, the utilization rate of the data is improved, and the cost is reduced.
2. The weight assignment algorithm is utilized to carry out weight matching, so that the importance degree of different indexes can be reflected, and the data can be fully utilized.
Drawings
FIG. 1 is a flow chart of a data processing method according to one embodiment of the present application;
FIG. 2 is a schematic flow chart of a sub-step of step S100 in FIG. 1;
FIG. 3 is a schematic flow chart of a sub-step of step S300 in FIG. 1;
FIG. 4 is a schematic flow chart of another substep of step S300 in FIG. 1;
FIG. 5 is a schematic flow chart showing a sub-step of step S340 in FIG. 4;
FIG. 6 is a schematic overall flow chart of a data processing method according to one embodiment of the present application;
FIG. 7 is a schematic diagram of a data processing apparatus according to one embodiment of the present application;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present specification, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present application, but not all embodiments.
In the description of embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described herein as "such as" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "or" for example "is intended to present related concepts in a concrete fashion.
In the description of the embodiments of the present application, the term "plurality" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating an indicated technical feature. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.
In the related art, the data uploading system automatically performs scoring according to key indexes of data, and the assignment of each index is formulated according to the existing historical data, for example: when the data A enters the database, the system can judge according to the set scoring indexes, automatically calculate a total scoring result, and the historical data is formulated to form indexes which cannot reflect the importance of different indexes, so that the data cannot be effectively utilized.
Based on the above, the embodiment of the application provides a data processing method, a device, an electronic device and a readable storage medium, wherein the data processing method quantizes qualitative data by acquiring a plurality of qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data, and each data can be assigned by quantization, so that the accuracy of data scoring is improved, the importance of the data is determined by the quantization levels, and a basis is provided for intelligent uploading of the data; dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level, so that the data can cover each quantization level, and the utilization rate of the data is improved; acquiring project data requirements, grading and distributing initial data volume by using a preset data grading model according to the project data requirements to obtain target data volume corresponding to each quantization level, and performing data distribution by using the data grading model according to the project requirements, so that the obtained target data volume can uniformly cover the data of each quantization level, the data utilization rate is improved, and the cost is reduced; and determining data to be uploaded according to each target data amount, and uploading the data to be uploaded to the project, so that automatic matching uploading can be realized. Compared with the prior art that the utilization rate of the item to the data is low, and the cost is increased, the method and the device have the advantages that the accuracy of score calculation is high, the utilization rate of the item to the data is improved, and therefore the cost is reduced. It should be noted that the data processing method can be applied to advertisement delivery, and the data uploaded for advertisement items are uniformly distributed on each level by grading and distributing the data, so that the cost is reduced according to the value of the uploaded data.
The technical scheme provided by the embodiment of the application is further described below with reference to the accompanying drawings.
Referring to fig. 1, fig. 1 is a flow chart of a data processing method according to an embodiment of the present application. The data processing method is applied to a data processing apparatus, and the data processing method is executed by a processor in an electronic device or a readable storage medium, and includes steps S100, S200, S300, and S400.
Step S100, a plurality of qualitative data are obtained, and the qualitative data are quantized to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data.
In one embodiment, qualitative data is collected in the form of a questionnaire and saved as text, and a plurality of qualitative data is read by a preset text reading function, which may be a txt () function. The qualitative data includes all data related to the project, and the qualitative data can be collected in the form of a questionnaire, which needs to be set, and questions in the questionnaire are set according to different projects to collect the required data. The data of the user logging platform can be collected, and the qualitative data can be obtained by collecting search records, purchase records, browsing records and evaluation of the user. And carrying out data preprocessing on the qualitative data, and preparing for subsequent data quantization. The data preprocessing includes performing deduplication processing, data cleaning processing, and the like on the data, which are not described herein.
As shown in fig. 2, the qualitative data is quantized to obtain a plurality of quantitative data, and quantization levels corresponding to the quantitative data, including, but not limited to, the following steps:
step S110, obtaining a preset grading index.
In some possible embodiments of the present application, the preset scoring index is empirically set by a professional, and the preset scoring index is stored in text form after being set, and is obtained by reading through a preset text reading function. The scoring indexes comprise whether a vehicle purchase plan exists, the type, surname summarization, labeling, whether a vehicle system exists or not, vehicle purchase time and whether contact is allowed or not, and whether the vehicle purchase plan exists comprises a clear vehicle purchase plan, a vehicle purchase plan after recovery and two times of undefined refusal (invalidation); types include success sheets and incompleteness sheets; surname summary includes surname and no surname; labeling includes preferred-established, preferred-contrasted, preferred-undetermined, first-in-view, recommended, first/recommended; the train with or without train comprises a train with or without train; the time of purchasing the commodity comprises one month, three months, half year, any time and one year; whether contact is allowed includes whether contact is allowed, text message or WeChat contact and ambiguous (or no query). And the scoring index is obtained to facilitate the subsequent scoring to obtain quantitative data and quantitative grade.
And step S120, configuring weights for the scoring indexes by using a preset weight assignment algorithm to obtain weight ratios.
In some possible embodiments of the present application, the weight assignment algorithm is one of a principal component analysis method, an AHP hierarchy analysis method, and a combined weighting method. Illustratively, the weight is configured on the scoring indexes by using a principal component analysis method to obtain the weight ratio, so that the importance degree of different indexes can be reflected, and the more important the scoring indexes are, the larger the weight ratio is. The AHP analytic hierarchy process is to decompose complex problems into various component factors, group the factors into hierarchical structures according to dominant relations, determine the relative importance of the factors by means of pairwise comparison, and reflect the importance of different indexes. The combined weighting method comprehensively obtains combined evaluation according to the objective weighting method and the subjective weighting method, and can reflect the importance degree of different indexes. And the weight ratio is obtained, so that the follow-up scoring is facilitated, and quantitative data and quantitative grades are obtained.
And step S130, scoring the qualitative data according to the weight ratio to obtain quantitative data, wherein the quantitative data is characterized as scoring values.
In some possible embodiments of the present application, the importance of different indexes can be reflected according to the weight ratio obtained in step S120, and a plurality of qualitative data is scored according to the weight ratio, so as to obtain a plurality of quantitative data, where the quantitative data is characterized as a scoring value. For example, a piece of qualitative data is scored, automatic scoring is performed, the weight score is higher than the score of the corresponding data, and the score is fully 11, 9, 8, etc. can be scored. By obtaining quantitative data, qualitative data can be used, so that the accuracy of scoring is higher.
Step S140, obtaining the quantization level based on the preset level score segment and the quantitative data.
In some possible embodiments of the present application, the preset grade score segments are set by a professional according to experience, and the preset grade score segments include 5 score segments, which are respectively (6.0,6.9), (7.0,7.9), (8.0,8.9), (9.0,9.9), (10.0, 11.0), and the corresponding grades are respectively grade E, grade D, grade C, grade B, and grade a. And obtaining a quantization grade corresponding to the quantitative data according to the preset grade score section and the quantitative data. For example, in the case that the score of the quantitative data is 9.5, the corresponding quantization level is B level, so that the quantization level corresponding to each piece of data can be obtained, and the higher the corresponding score segment, the higher the level is, which is beneficial to the reasonable distribution of the data. The higher the level is, the higher the cost is when the corresponding data is used, so the application can make the used data uniformly distributed on each level as much as possible, improve the data use rate and reduce the cost.
Step S200, dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantitative levels to obtain initial data quantity corresponding to each quantitative level.
In an embodiment, according to the number of the quantitative data and the number of the quantization levels, for example, there are 5 quantization levels, and the quantitative data is divided equally according to the 5 quantization levels, so as to obtain an initial data amount corresponding to each quantization level. The quantitative data may be divided in other proportions than the average division, i.e., the initial data amount for each level may be different. The method is favorable for the follow-up optimal allocation according to the initial data quantity, so that the data utilization rate is higher, and the cost is saved.
Step S300, acquiring project data requirements, and grading and distributing initial data volume by using a preset data grading model according to the project data requirements to obtain target data volume corresponding to each quantization level.
In an embodiment, the project data requirement is a requirement of data required by a specific project, which is determined according to the nature of the project, and the project data requirement determined according to the nature of the project is acquired through a preset data acquisition interface. The data can be distributed according to the requirements in the follow-up process by acquiring the project data requirements, and the use efficiency of the data is improved. And then, grading and distributing the initial data volume by utilizing a preset data grading model according to project data requirements to obtain target data volumes corresponding to all quantization levels, so that the data distribution is realized, the target data volumes can uniformly cover the data of all quantization levels, and the data utilization rate is improved.
In an embodiment, in the case that the project data requirement is a data requirement corresponding to a target project, the project data requirement includes a first target uploading amount and a first target effective rate, as shown in fig. 3, according to the project data requirement, the initial data amount is scored and allocated by using a preset data scoring model to obtain target data amounts corresponding to each quantization level, including but not limited to the following steps:
step S310, performing equation construction on the initial data volume, the first target uploading volume, the first target effective rate and the quantization level by using a preset data scoring model to obtain a first equation.
In some possible embodiments of the present application, the first target uploading amount is a required amount of a current data amount of a single item, the first target effective rate is a duty ratio of effective data in the current data amount, specifically, according to the current data amount, user usage data is effective data, the number of the effective data is counted, and the first target effective rate is obtained by calculating a ratio of the number of the effective data to the current data amount. And constructing an equation by using a preset data scoring model to obtain a first equation, wherein the first equation comprises the initial data quantity, the first target uploading quantity, the first target effective rate and the quantization level. Because the quantization levels comprise an A level, a B level, a C level, a D level and an E level, the equation construction is carried out on the initial data quantity, the first target uploading quantity, the first target effective rate and the quantization level by utilizing a preset data scoring model, and a first equation is obtained, wherein the first equation is expressed as follows:
Where N is denoted as the first target upload, r is denoted as the first target effective rate,all indicated as initial data amount corresponding to each quantization level,/-, respectively>And the weight ratio corresponding to each quantization level is shown. By constructing the first equation, the distribution of data in each level can be reflected.
Step S320, solving the first equation to obtain a solution of the equation, and optimizing the solution of the equation to obtain the target data quantity corresponding to each quantization level.
In some possible embodiments of the present application, the solution of the equation constructed according to step S310 is obtained by solving the solution of the equation, where the solution of the equation is expressed as:
in other possible embodiments of the present application,according to the above inequalityCan get +.>Is a value range of>Representing its range of values, the formula can be expressed as:
in some other possible embodiments of the present application, the solution of the equation is optimized, so that high-level data is as few as possible, low-level data is as many as possible, and the target data amount corresponding to each quantization level is obtained, so that corresponding data can be provided according to the project requirement, the provided data can uniformly cover the data of each quantization level, the data utilization rate is improved, and the cost is reduced. It should be noted that, the combination of the obtained target data amounts may have a plurality of combination modes, and then the data to be uploaded is determined according to each target data amount, and one combination mode is selected so as to maximize the utilization rate of the data.
Step S400, determining data to be uploaded according to each target data amount, and uploading the data to be uploaded to the project.
In one embodiment, in order to maximize the utilization rate of different types of data, the data to be uploaded is determined according to each target data amount, and then the utilization rate of the data can be improved by considering that a preset utilization rate function is maximized within a feasible range. Illustratively, the preset utilization function is expressed as:
wherein a, b, c, d, e are all values greater than 0 and are adjustable parameters,the total data quantity expressed as each quantization level at present can improve the value of the corresponding parameter of a certain type under the condition that the utilization rate of the certain type needs to be increased. By calculating the utilization rateAnd the function is used for enabling the data to be uploaded, which corresponds to the maximum utilization rate, to come from each quantization level. And determining the data to be uploaded by maximizing the utilization rate function, and uploading the data to be uploaded to the project for use.
In another embodiment, in the case that the utilization rate reaches the maximum, the cost of the data to be uploaded, which is determined according to each target data amount, is also minimum. Illustratively, the cost of each of the A-level, B-level, C-level, D-level, and E-level data can also be optimized for economic cost optimization, expressed as Optimizing to enable a preset cost function to be at the minimum value of a feasible range, wherein the preset cost function is expressed as:
the data to be uploaded is selected from the target data volume, and is uploaded to the project for use, and the data to be uploaded can cover a plurality of quantization levels, so that the utilization rate of the project on the data is improved, the utilization rate of the data is high, and the cost is minimum.
In an embodiment, in the case that the project data requirement is a data requirement corresponding to a plurality of target projects, the project data requirement includes a second target uploading amount corresponding to each target project and a second target effective rate, as shown in fig. 4, according to the project data requirement, the initial data amount is scored and allocated by using a preset data scoring model to obtain a target data amount corresponding to each quantization level, including but not limited to the following steps:
and S330, constructing an equation of the initial data volume, the second target uploading volume, the second target effective rate and the quantization level by using a preset data scoring model to obtain a second equation and a conditional constraint inequality, wherein the conditional constraint inequality is obtained by determining the initial data volume corresponding to the quantization level and the second target uploading volume corresponding to each target item.
In some possible embodiments of the present application, in a case where the project data requirement is a data requirement corresponding to a plurality of target projects, that is, the target projects are a plurality of target projects, the target projects may be shopping projects. The second target uploading amount is the required amount of the current data amount of each target item, the second target effective rate is the duty ratio of effective data in the current data amount, specifically, according to the current data amount, the user use data are effective data, the quantity of the effective data is counted, and the second target effective rate is obtained by calculating the ratio of the quantity of the effective data to the current data amount. And constructing an equation by using a preset data scoring model to obtain a second equation and a conditional constraint inequality, wherein the conditional constraint inequality is obtained by determining the initial data quantity corresponding to the quantization level and the second target uploading quantity corresponding to each target item.
The second equation is expressed as:
wherein,the +.>Second target upload amount corresponding to each target item,/->The +.>Second target effective rate corresponding to the target item, < > >All indicated as initial data amount corresponding to each quantization level,/-, respectively>Are each represented as corresponding to each quantization levelWeight ratio. The distribution of the data of each item in each level can be reflected by constructing a second equation.
The total data amount of each quantization level is respectively:the conditional constraint inequality is expressed as:
and step S340, obtaining a solution set corresponding to the target item according to the second equation and the conditional constraint inequality, and calculating the optimal ratio based on the solution set to obtain the target data volume.
In some possible embodiments of the present application, the solution set corresponding to the target item is obtained by solving the second equation and the conditional constraint inequality constructed in step S330, and is expressed as. The solution set may be expressed as:. Other solution sets corresponding to each quantization level may also be calculated, similar to the above manner, and will not be described herein.
In other possible embodiments of the present application, the optimal matching is calculated based on the solution set, so that high-level data is as few as possible, low-level data is as many as possible, and the target data amount corresponding to each quantization level is obtained, so that corresponding data is provided according to the project data requirement, the data provided for each project can uniformly cover the data of each quantization level, the data utilization rate is improved, and the cost is reduced.
As shown in fig. 5, the optimal matching is calculated based on the solution set to obtain the target data amount, including but not limited to the following steps:
step S341, calculating to obtain a target optimization function according to the solution values in the solution set, wherein the target optimization function comprises data quality and data utilization rate.
In some possible embodiments of the present application, the objective optimization function includes a data quality and a data usage rate, wherein the data quality is a user satisfaction with the data, the data quality satisfaction is calculated from solution values in the solution set, and the satisfaction function is calculated by using a preset satisfaction functionExpressed as:
wherein the method comprises the steps ofAre parameters formulated according to the data volume.
Specifically, the data usage rate refers to the usage rate of data in the target data amount, the data usage rate is calculated by using a preset usage rate functionExpressed as:
specifically, a target optimization function is calculated according to the calculated data quality or satisfaction and the data utilization rate, and the target optimization function is expressed asThe formula is +.>: . And the calculation of the target optimization function is beneficial to the subsequent optimization of the solution set, so that the target data size is obtained.
And S342, optimizing the solution set by using a target optimization function to obtain a target data volume.
In some possible embodiments of the present application, in order to make the data quality higher, the data usage rate is higher, and the solution set is optimized by the maximizing objective optimization function, so as to obtain the objective data amount, and improve the data usage rate.
In an embodiment, the data to be uploaded is determined according to each target data amount, and optimization is performed so that the data utilization rate of different types reaches the maximum, and then the maximization of the preset utilization rate function in the feasible range can be considered, and the calculation of the maximum utilization rate of each target item in the plurality of target items is similar to the above, and is not repeated here.
In another embodiment, the objective optimization function is optimized, and the data to be uploaded is determined from each objective data volume, so that the utilization rate of the data is high, and the cost is low due to the high utilization rate of the data. Illustratively, the cost of each piece of data corresponding to the A-level, B-level, C-level, D-level, and E-level is expressed asAnd optimizing to enable the preset cost function to be at the minimum value of the feasible range, so as to obtain the minimum cost. Cost function optimization is similar to that described above for a project data requirement and will not be described in detail herein.
Illustratively, assuming a total number of data per day, qualitative data is n=20000 pieces, where each type of data in class a, class B, class C, class D, and class E is 4000 pieces. And acquiring target data quantity and target effective rate of project data requirements, and optimizing data distribution of each level according to the target data quantity and the target effective rate.
In the case of 7 items, namely 7 items are required, the obtained target data amount and the target effective rate are as follows:
target item number 1 2 3 4 5 6 7
Target upload volume 1000 2000 3000 4000 5000 1300 700
Data efficiency 0.3 0.3 0.3 0.3 0.3 0.3 0.3
Calculation of the parameter wi= (n-i)/n,/>,/>,/>. a, b, c, d, e are 10,8,6,4,1, respectively. The data to be uploaded of each item is obtained according to the item requirements, different quantization levels are covered by the data to be uploaded of each item, the utilization rate of the data can be improved, and the cost is reduced.
The data to be uploaded are distributed as follows:
target item number 1 2 3 4 5 6 7
A data duty cycle 56.6% 33.7% 5.7% 0 28.8% 44.3% 0
B data duty cycle 0 0 40.0% 42.7% 0.1% 0 66.6%
C data duty cycle 0 0 0 36.0% 0 0 0.1%
D data duty cycle 0.3% 48.2% 54.3% 0 70.3% 0.7% 0
E data duty cycle 43.1% 18.1% 0 21.3% 0.8% 55% 33.3%
According to the data distribution table to be uploaded, most of data can be covered on a plurality of levels, the importance of the data can be reflected according to the distribution of the data, the data is proportioned according to the requirement of project data according to the importance, the effective utilization rate of the data is improved, the data utilization rate can be improved by uploading the data to the project through the distribution, and further the cost is reduced.
As shown in fig. 6, the embodiment of the application provides a schematic diagram of an arrangement flow of a data processing method, firstly, qualitative data is obtained, the qualitative data is quantized to obtain quantitative data, the quantitative data is stored in a database, and the quantitative data is recorded for convenient subsequent use and viewing; the method comprises the steps of obtaining project data requirements, grading and distributing quantitative data stored in a database by utilizing a big data grading model according to the project data requirements to obtain target data quantity, determining data to be uploaded according to the target data quantity, and uploading the data to be uploaded to a project, so that data of one project or a plurality of projects can be provided. And the quantitative data can be directly scored and distributed by utilizing a big data scoring model according to the project data requirement, the target data quantity is obtained, the data to be uploaded is determined according to the target data quantity, the data to be uploaded is uploaded to the project, and the data of one project or a plurality of projects can be provided.
As shown in fig. 7, an embodiment of the present application provides a data processing apparatus 100, where a data acquisition module 110 of the apparatus 100 acquires a plurality of qualitative data, quantizes the qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data, and performs quantization to assign a value to each day of data, so as to improve accuracy of data scoring, and determines importance of the data according to the quantization levels, so as to provide a basis for intelligent uploading of the data; then, the data dividing module 120 is utilized to divide the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels, so as to obtain initial data quantity corresponding to each quantization level, so that the data can cover each quantization level, and the utilization rate of the data is improved; then, the scoring module 130 is utilized to acquire project data requirements, a preset data scoring model is utilized to score and distribute initial data volume according to the project data requirements, target data volume corresponding to each quantization level is obtained, and the data scoring model is utilized to distribute data according to the project requirements, so that the obtained target data volume can uniformly cover the data of each quantization level, the data utilization rate is improved, and the cost is reduced; finally, the data uploading module 140 determines the data to be uploaded according to each target data amount, and uploads the data to be uploaded to the project, so that automatic matching uploading can be realized.
It should be noted that, the data obtaining module 110 is connected to the data dividing module 120, the data dividing module 120 is connected to the scoring module 130, and the scoring module 130 is connected to the data uploading module 140. The data processing method is applied to the data processing device 100, the data processing device 100 quantizes qualitative data to obtain quantitative data and quantized levels, the quantitative data corresponding to each quantized level is scored and distributed according to project data requirements after the quantitative data and quantized levels are divided according to the quantized levels, a target data amount is obtained, and the technical means of uploading data is determined according to the target data amount, so that the uploaded data can cover different levels, the utilization rate of the data is improved, and the cost is reduced.
Also to be described is: in the device provided in the above embodiment, when implementing the functions thereof, only the division of the above functional modules is used as an example, in practical application, the above functional allocation may be implemented by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to implement all or part of the functions described above. In addition, the embodiments of the apparatus and the method provided in the foregoing embodiments belong to the same concept, and specific implementation processes of the embodiments of the method are detailed in the method embodiments, which are not repeated herein.
The application also discloses electronic equipment. Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to the disclosure of the embodiment of the present application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, at least one communication bus 502.
Wherein a communication bus 502 is used to enable connected communications between these components.
The user interface 503 may include a Display screen (Display) and a Camera (Camera), and the optional user interface 503 may further include a standard wired interface and a standard wireless interface.
The network interface 504 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), among others.
Wherein the processor 501 may include one or more processing cores. The processor 501 connects various parts throughout the server using various interfaces and lines, performs various functions of the server and processes data by executing or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and invoking data stored in the memory 505. Alternatively, the processor 501 may be implemented in hardware in at least one of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), programmable logic array (Programmable Logic Array, PLA). The processor 501 may integrate one or a combination of several of a central processing unit (Central Processing Unit, CPU), an image processor (Graphics Processing Unit, GPU), and a modem, etc. The CPU mainly processes an operating system, a user interface, an application program and the like; the GPU is used for rendering and drawing the content required to be displayed by the display screen; the modem is used to handle wireless communications. It will be appreciated that the modem may not be integrated into the processor 501 and may be implemented by a single chip.
The Memory 505 may include a random access Memory (Random Access Memory, RAM) or a Read-Only Memory (Read-Only Memory). Optionally, the memory 505 comprises a non-transitory computer readable medium (non-transitory computer-readable storage medium). Memory 505 may be used to store instructions, programs, code sets, or instruction sets. The memory 505 may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-described various method embodiments, etc.; the storage data area may store data or the like involved in the above respective method embodiments. The memory 505 may also optionally be at least one storage device located remotely from the processor 501. Referring to fig. 8, an operating system, a network communication module, a user interface module, and an application program of a data processing method may be included in the memory 505 as a computer storage medium.
In the electronic device 500 shown in fig. 8, the user interface 503 is mainly used for providing an input interface for a user, and acquiring data input by the user; and the processor 501 may be configured to invoke an application program in the memory 505 that stores a data processing method, which when executed by the one or more processors 501, causes the electronic device 500 to perform the method as in one or more of the embodiments described above. It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
In the several embodiments provided herein, it should be understood that the disclosed apparatus may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, such as a division of units, merely a division of logic functions, and there may be additional divisions in actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some service interface, device or unit indirect coupling or communication connection, electrical or otherwise.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a memory, including several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of the embodiments of the present application. And the aforementioned memory includes: various media capable of storing program codes, such as a U disk, a mobile hard disk, a magnetic disk or an optical disk.
The above are merely exemplary embodiments of the present disclosure and are not intended to limit the scope of the present disclosure. That is, equivalent changes and modifications are contemplated by the teachings of this disclosure, which fall within the scope of the present disclosure. Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure.
This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a scope and spirit of the disclosure being indicated by the claims.

Claims (9)

1. A method of data processing, the method comprising:
obtaining a plurality of qualitative data, and quantizing the qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data;
dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level;
acquiring project data requirements, and grading and distributing the initial data volume by using a preset data grading model according to the project data requirements to obtain target data volumes corresponding to the quantization levels;
determining data to be uploaded according to the target data amount, and uploading the data to be uploaded to a project;
under the condition that the project data requirement is a data requirement corresponding to a target project, the project data requirement comprises a first target uploading amount and a first target effective rate;
And grading and distributing the initial data volume by using a preset data grading model according to the project data requirement to obtain target data volumes corresponding to the quantization levels, wherein the grading and distributing comprises the following steps:
carrying out equation construction on the initial data quantity, the first target uploading quantity, the first target effective rate and the quantization level by using a preset data scoring model to obtain a first equation;
and solving according to the first equation to obtain a solution of an equation, and optimizing the solution of the equation to obtain the target data quantity corresponding to each quantization level.
2. The method of claim 1, wherein said quantizing said qualitative data to obtain a plurality of quantitative data, and a quantization level corresponding to each of said quantitative data, comprises:
acquiring a preset scoring index;
configuring weights for the scoring indexes by using a preset weight assignment algorithm to obtain weight ratios;
scoring the qualitative data according to the weight ratio to obtain quantitative data, wherein the quantitative data is characterized as scoring values;
and obtaining the quantization grade based on a preset grade score section and the quantitative data.
3. The method of claim 2, wherein the weight assignment algorithm is one of a principal component analysis method, an AHP hierarchy analysis method, and a combined weighting method.
4. The method of claim 1, wherein the quantization levels include a level, B level, C level, D level, E level;
the equation construction is performed on the initial data volume, the first target uploading volume, the first target effective rate and the quantization level by using a preset data scoring model to obtain a first equation, including:
the first mode is as follows:
x A +x B +x C +x D +x E =N
p A x A +p B x B +p C x C +p D x D +p E x E =rN
wherein N is represented as the first target uploading amount, r is represented as the first target effective rate, and x A 、x B 、x C 、x D 、x E Are each represented as the initial data amount, p, corresponding to each of the quantization levels A 、p B 、p C 、p D 、p E And the weight ratio corresponding to each quantization level is shown.
5. The method of claim 1, wherein in the event that the project data requirement is a data requirement corresponding to a plurality of target projects, the project data requirement includes a second target upload amount and a second target effective rate corresponding to each of the target projects;
and grading and distributing the initial data volume by using a preset data grading model according to the project data requirement to obtain target data volumes corresponding to the quantization levels, wherein the grading and distributing comprises the following steps:
Carrying out equation construction on the initial data volume, the second target uploading volume, the second target effective rate and the quantization level by using a preset data scoring model to obtain a second equation and a conditional constraint inequality, wherein the conditional constraint inequality is obtained by determining the initial data volume corresponding to the quantization level and the second target uploading volume corresponding to each target item;
and obtaining a solution set corresponding to the target item according to the second equation and the condition constraint inequality, and calculating an optimal ratio based on the solution set to obtain the target data volume.
6. The method of claim 5, wherein calculating an optimal ratio based on the solution set to obtain the target data amount comprises:
according to the solution numerical value in the solution set, calculating to obtain a target optimization function, wherein the target optimization function comprises data quality and data utilization rate;
and optimizing the solution set by using the target optimization function to obtain the target data volume.
7. A data processing apparatus, the apparatus comprising:
the data acquisition module (110) is used for acquiring a plurality of qualitative data, quantizing the qualitative data to obtain a plurality of quantitative data and quantization levels corresponding to the quantitative data;
The data dividing module (120) is used for dividing the quantitative data according to the quantity of the quantitative data and the quantity of the quantization levels to obtain initial data quantity corresponding to each quantization level;
the scoring module (130) is used for acquiring project data requirements, and scoring and distributing the initial data volume by utilizing a preset data scoring model according to the project data requirements to obtain target data volumes corresponding to the quantization levels;
the data uploading module (140) is used for determining data to be uploaded according to the target data quantity and uploading the data to be uploaded to a project;
under the condition that the project data requirement is a data requirement corresponding to a target project, the project data requirement comprises a first target uploading amount and a first target effective rate; the scoring module (130) is specifically configured to perform equation construction on the initial data volume, the first target uploading volume, the first target effective rate and the quantization level by using a preset data scoring model to obtain a first equation; and solving according to the first equation to obtain a solution of an equation, and optimizing the solution of the equation to obtain the target data quantity corresponding to each quantization level.
8. An electronic device comprising a processor (501), a memory (505), a user interface (503), a communication bus (502) and a network interface (504), the processor (501), the memory (505), the user interface (503) and the network interface (504) being respectively connected to the communication bus (502), the memory (505) being adapted to store instructions, the user interface (503) and the network interface (504) being adapted to communicate to other devices, the processor (501) being adapted to execute the instructions stored in the memory (505) to cause the electronic device (500) to perform the method according to any of claims 1-6.
9. A computer readable storage medium storing instructions which, when executed, perform the method of any one of claims 1-6.
CN202311509838.9A 2023-11-14 2023-11-14 Data processing method, device, electronic equipment and readable storage medium Active CN117236801B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311509838.9A CN117236801B (en) 2023-11-14 2023-11-14 Data processing method, device, electronic equipment and readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311509838.9A CN117236801B (en) 2023-11-14 2023-11-14 Data processing method, device, electronic equipment and readable storage medium

Publications (2)

Publication Number Publication Date
CN117236801A CN117236801A (en) 2023-12-15
CN117236801B true CN117236801B (en) 2024-01-26

Family

ID=89084604

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311509838.9A Active CN117236801B (en) 2023-11-14 2023-11-14 Data processing method, device, electronic equipment and readable storage medium

Country Status (1)

Country Link
CN (1) CN117236801B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109299085A (en) * 2018-11-07 2019-02-01 平安医疗健康管理股份有限公司 A kind of data processing method, electronic equipment and storage medium
CN111861227A (en) * 2020-07-23 2020-10-30 中国平安财产保险股份有限公司 User recommendation method and device based on membership analysis, electronic equipment and medium
CN114021873A (en) * 2021-09-23 2022-02-08 上海仪电人工智能创新院有限公司 Data index quantification method and intelligent park enterprise value evaluation system
CN116932549A (en) * 2023-07-21 2023-10-24 企知道科技有限公司 Intelligent model-based platform data storage method, system, medium and equipment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4258174A1 (en) * 2022-04-07 2023-10-11 Aptiv Technologies Limited Methods and systems for determining a representative input data set for post-training quantization of artificial neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109299085A (en) * 2018-11-07 2019-02-01 平安医疗健康管理股份有限公司 A kind of data processing method, electronic equipment and storage medium
CN111861227A (en) * 2020-07-23 2020-10-30 中国平安财产保险股份有限公司 User recommendation method and device based on membership analysis, electronic equipment and medium
CN114021873A (en) * 2021-09-23 2022-02-08 上海仪电人工智能创新院有限公司 Data index quantification method and intelligent park enterprise value evaluation system
CN116932549A (en) * 2023-07-21 2023-10-24 企知道科技有限公司 Intelligent model-based platform data storage method, system, medium and equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于决策融合的多卫星通信系统态势分析技术;董彦磊;河北工业科技;第39卷(第4期);正文第321-326页 *

Also Published As

Publication number Publication date
CN117236801A (en) 2023-12-15

Similar Documents

Publication Publication Date Title
CN104834731B (en) A kind of recommended method and device from media information
CN106952159B (en) Real estate collateral risk control method, system and storage medium
CN108364085A (en) A kind of take-away distribution time prediction technique and device
CN110298547A (en) Methods of risk assessment, device, computer installation and storage medium
CN107729519B (en) Multi-source multi-dimensional data-based evaluation method and device, and terminal
CN112612942B (en) Social big data-based fund recommendation system and method
CN111489201A (en) Method, device and storage medium for analyzing customer value
CN109615504A (en) Products Show method, apparatus, electronic equipment and computer readable storage medium
CN111833018A (en) Patent analysis method and system for science and technology project
CN109428910A (en) A kind of data processing method, apparatus and system
CN115170294A (en) Client classification method and device and server
CN113222326A (en) Method and device for evaluating maturity of scientific and technological resource service platform
CN113361980A (en) Big data asset value evaluation system and method
CN112435047A (en) Marketing outbound data recommendation method based on buried point data
CN111626767A (en) Resource data distribution method, device and equipment
Sidorov et al. Extended innovation diffusion models and their empirical performance on real propagation data
von Rimscha It’s not the economy, stupid! External effects on the supply and demand of cinema entertainment
CN114154672A (en) Data mining method for customer churn prediction
CN117236801B (en) Data processing method, device, electronic equipment and readable storage medium
CN111311310B (en) Advertisement order pushing method and device, storage medium and electronic device
CN112232945B (en) Method and device for determining personal client credit
CN113407827A (en) Information recommendation method, device, equipment and medium based on user value classification
US20140278739A1 (en) Methods and apparatus to monitor products in stores
CN111698332A (en) Method, device and equipment for distributing business objects and storage medium
KR20180087198A (en) Apparatus for providing contents using number information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant