CN108764273A - A kind of method, apparatus of data processing, terminal device and storage medium - Google Patents
A kind of method, apparatus of data processing, terminal device and storage medium Download PDFInfo
- Publication number
- CN108764273A CN108764273A CN201810309823.0A CN201810309823A CN108764273A CN 108764273 A CN108764273 A CN 108764273A CN 201810309823 A CN201810309823 A CN 201810309823A CN 108764273 A CN108764273 A CN 108764273A
- Authority
- CN
- China
- Prior art keywords
- branch mailbox
- sample data
- decision tree
- feature
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 56
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000003860 storage Methods 0.000 title claims abstract description 17
- 238000003066 decision tree Methods 0.000 claims abstract description 175
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 34
- 230000008878 coupling Effects 0.000 claims description 20
- 238000010168 coupling process Methods 0.000 claims description 20
- 238000005859 coupling reaction Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 16
- 238000012360 testing method Methods 0.000 claims description 12
- 230000001737 promoting effect Effects 0.000 claims description 3
- 238000000605 extraction Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 21
- 230000008569 process Effects 0.000 description 12
- 238000009826 distribution Methods 0.000 description 7
- 238000012549 training Methods 0.000 description 7
- 238000005520 cutting process Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000002708 enhancing effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- VYZAMTAEIAYCRO-UHFFFAOYSA-N Chromium Chemical compound [Cr] VYZAMTAEIAYCRO-UHFFFAOYSA-N 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000009975 flexible effect Effects 0.000 description 1
- 239000012535 impurity Substances 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of method, apparatus of data processing, terminal device and storage medium, the method includes：Obtain configuration information and initial sample data；Configuration file is generated according to the configuration information；Branch mailbox is carried out to initial sample data according to the branch mailbox configuration information in configuration file, and effective sample data are obtained based on final branch mailbox result；One-hot coding is carried out to effective sample data, and digitized samples collection is built according to the result of one-hot coding；Decision Tree algorithms are promoted to digitlization sample set application gradient, generate decision tree；The feature that path using every decision tree includes carries out the model prediction of Logic Regression Models using assemblage characteristic as assemblage characteristic.Technical scheme of the present invention realizes the automatic accurate extraction to the assemblage characteristic of initial sample data, to when carrying out model prediction according to the assemblage characteristic, effectively improve the accuracy of model prediction.
Description
Technical field
The present invention relates to field of computer technology more particularly to a kind of according to the method, apparatus of processing, terminal device and storage
Medium.
Background technology
In general, in various data analyses and modeling process, it is necessary first to carry out feature to the data characteristics of sample data
Processing.
Currently, the signature analysis of conventional machines study often relies on artificial experience and analyzes limited sample data,
And then determine characteristic processing algorithm, it takes time and effort, or single features Processing Algorithm is all made of to different data characteristicses.So
And the characteristic processing algorithm or single features Processing Algorithm that either artificial experience determines, it cannot meet data characteristics
Diversified feature, the characteristic processing result because obtained from tend not to accurately reflect the true feature of data characteristics, cause final
The prediction result accuracy rate of the model of structure is not high.
Invention content
The embodiment of the present invention provides a kind of method, apparatus of data processing, terminal device and storage medium, existing to solve
The result for carrying out characteristic processing in technology to data characteristics is inaccurate, the problem for causing model prediction result accuracy rate not high.
In a first aspect, the embodiment of the present invention provides a kind of method of data processing, including：
Configuration information is obtained, and initial sample data is obtained based on the configuration information；
According to the configuration information, configuration file is generated according to preset configuration template；
The branch mailbox configuration information in the configuration file is obtained, according to the branch mailbox configuration information to the initial sample number
According to progress branch mailbox, and the initial sample data is handled based on final branch mailbox result, obtains effective sample data,
In, the effective sample data include the branch mailbox characteristic value of branch mailbox feature；
One-hot coding is carried out to the effective sample data, and digitized samples are built according to the result of the one-hot coding
Collection；
Decision Tree algorithms are promoted to the digitized samples collection application gradient, generate the decision tree mould for including n decision tree
Type, wherein n is the positive integer more than 1；
The feature that path using every decision tree in the decision-tree model includes uses the combination as assemblage characteristic
Feature carries out the model prediction of Logic Regression Models.
Second aspect, the embodiment of the present invention provide a kind of device of data processing, including：
Data acquisition module obtains initial sample data for obtaining configuration information, and based on the configuration information；
File generating module, for according to the configuration information, configuration file to be generated according to preset configuration template；
Branch mailbox module, for obtaining the branch mailbox configuration information in the configuration file, according to the branch mailbox configuration information pair
The initial sample data carries out branch mailbox, and is handled the initial sample data based on final branch mailbox result, is had
Imitate sample data, wherein the effective sample data include the branch mailbox characteristic value of branch mailbox feature；
Digital module, for carrying out one-hot coding to the effective sample data, and according to the knot of the one-hot coding
Fruit builds digitized samples collection；
Decision tree builds module, and for promoting decision Tree algorithms to the digitized samples collection application gradient, it includes n to generate
The decision-tree model of decision tree, wherein n is the positive integer more than 1；
Model prediction module, the feature for including using the path of every decision tree in the decision-tree model is as combination
Feature carries out the model prediction of Logic Regression Models using the assemblage characteristic.
The third aspect, the embodiment of the present invention provide a kind of terminal device, including memory, processor and are stored in described
In memory and the computer program that can run on the processor, the processor are realized when executing the computer program
The step of method of the data processing.
Fourth aspect, the embodiment of the present invention provide a kind of computer readable storage medium, the computer-readable storage medium
The step of matter is stored with computer program, and the computer program realizes the method for the data processing when being executed by processor.
In the method, apparatus of data processing provided in an embodiment of the present invention a kind of, terminal device and storage medium, in basis
Configuration information obtains initial sample data, and after generating configuration file according to preset configuration template, obtains configuration file first
In branch mailbox configuration information, according to the branch mailbox configuration information to initial sample data carry out branch mailbox, and be based on final branch mailbox result
Initial sample data is handled, obtains effective sample data, then one-hot coding is carried out to effective sample data, and according to only
The result of heat coding builds digitized samples collection, then promotes decision Tree algorithms to digitlization sample set application gradient, generates packet
Decision-tree model containing more decision trees, the feature for including using the path of every decision tree in decision-tree model finally is as combination
Feature, using the assemblage characteristic carry out Logic Regression Models model prediction, by initial sample data is carried out successively branch mailbox,
One-hot coding and decision tree structure, realization carry out initial sample data automatic accurate feature extraction so that obtained combination
Feature can accurately reflect the data characteristics of initial sample data, thus when carrying out model prediction according to the assemblage characteristic, energy
Enough effectively improve the accuracy of model prediction.
Description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the present invention
Example, for those of ordinary skill in the art, without having to pay creative labor, can also be according to these attached drawings
Obtain other attached drawings.
Fig. 1 is the flow chart of the method for the data processing provided in the embodiment of the present invention 1；
Fig. 2 be the data processing provided in the embodiment of the present invention 1 method in regression tree simple examples figure；
Fig. 3 be the data processing provided in the embodiment of the present invention 1 method according to data describe complete configuration file
The flow chart automatically updated；
Fig. 4 be the data processing provided in the embodiment of the present invention 1 method in step S4 flow chart；
Fig. 5 be the data processing provided in the embodiment of the present invention 1 method in digitlization sample set in digitlization sample
The flow chart of this progress across variable coding；
Fig. 6 be the data processing provided in the embodiment of the present invention 1 method in step S6 flow chart；
Fig. 7 is the schematic diagram of the device of the data processing provided in the embodiment of the present invention 2；
Fig. 8 is the schematic diagram of the terminal device provided in the embodiment of the present invention 4.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, the every other implementation that those of ordinary skill in the art are obtained without creative efforts
Example, shall fall within the protection scope of the present invention.
Embodiment 1
Referring to Fig. 1, Fig. 1 shows the implementation process of the method for data processing provided in this embodiment.The data processing
Method for realizing the structure of feature coding model, and can be applied to the prediction to Logic Regression Models.Details are as follows：
S1：Configuration information is obtained, and initial sample data is obtained based on the configuration information.
In embodiments of the present invention, configuration information includes the parameter information for modeling required various parameters, specifically be may include
The resource location information of initial sample data, the configuration parameter of null value filling information, branch mailbox configuration information and decision-tree model
Deng.Configuration information can by user pre-setting according to application.
Specifically, pre-set configuration information is obtained, the resource-niche of initial sample data is extracted from the configuration information
Confidence ceases, and obtains corresponding initial sample data according to the resource location information.
S2：According to configuration information, configuration file is generated according to preset configuration template.
Specifically, the configuration information obtained according to step S1 generates corresponding configuration file according to preset configuration template.
It should be noted that the configuration parameter needed for different models may be different, the structure definition of configuration file also may be used
Can be different, for the modeling requirement of different models, the corresponding configuration template of each model is pre-set, to according to mould to be built
Type selects corresponding configuration template to obtain corresponding configuration from configuration information according to the configuration parameter requirements of the configuration template
Parameter information, and generate corresponding configuration file according to the file structure of the configuration template so that the subsequently energy in modeling process
Required configuration parameter is read in the slave configuration file of enough fast and flexibles.
Configuration file can be extensible markup language (Extensible Markup Language, xml) file, also may be used
To be the file of extended formatting, it is not limited herein.
Wherein, xml document is a kind of original language file that permission user is defined the markup language of oneself, is processing
The effective tool of distributed frame information additionally provides a kind of tree-like hierarchical structure in xml document, and it is quickly fixed to may be implemented
Position.
S3：The branch mailbox configuration information in configuration file is obtained, initial sample data is carried out according to the branch mailbox configuration information
Branch mailbox, and initial sample data is handled based on final branch mailbox result, obtain effective sample data, wherein effective sample
Data include the branch mailbox characteristic value of branch mailbox feature.
In embodiments of the present invention, branch mailbox configuration information includes branch mailbox feature and case number threshold value etc., wherein branch mailbox is characterized as
It is maximum branch mailbox quantity to need the characteristic attribute of progress branch mailbox, such as age, case number threshold value.
Branch mailbox configuration information can be flexibly arranged by user according to modeling requirement or using needs.
Specifically, according to branch mailbox feature and case number threshold value, branch mailbox is carried out to the initial sample data that step S1 is obtained, is obtained
Final branch mailbox is as a result, the final branch mailbox result includes the branch mailbox number of branch mailbox feature and the branch mailbox characteristic value per case.Then, being based on should
Final branch mailbox result determines the branch mailbox characteristic value of the branch mailbox feature of initial sample data, obtains the effective sample for including branch mailbox characteristic value
Notebook data.
It should be noted that branch mailbox process can be based on spark distributed computing frameworks, complete to initial sample data
Automatic branch mailbox can rapidly and accurately carry out feature extraction while utmostly preserving raw sample data information, realize
Rapid modeling.
By taking branch mailbox is characterized as the age as an example, if final branch mailbox result be [10,35), [35,45) and [45,80] totally three case,
Then according to the final branch mailbox result, it is assumed that the age of certain initial sample data is 20 years old, then the branch mailbox of the initial sample data is special
The branch mailbox characteristic value of sign be [10,35), i.e., the branch mailbox of branch mailbox feature is special in the corresponding effective sample data of the initial sample data
Value indicative be [10,35).
S4：One-hot coding is carried out to effective sample data, and digitized samples collection is built according to the result of one-hot coding.
In embodiments of the present invention, one-hot coding, that is, one-hot codings, also known as an efficient coding, principle are to make
N number of state is encoded with N bit status registers, each state has independent register-bit, and when arbitrary
Only one effectively.
Specifically, for each feature of effective sample data, if it has M different characteristic values, according to one-
Hot codings obtain M binary feature.Also, these characteristic value mutual exclusions are only activated there are one characteristic value, are activated every time
Characteristic value be set as 1, remaining characteristic value not being activated then is set to constant 0, finally obtain feature each characteristic value correspond to
Fundamental digital coding.
It is encoded according to fundamental digital, each feature of each effective sample data is encoded, each feature is obtained
Digital coding, then the digital coding of whole features is combined, obtain the corresponding digitlization of each effective sample data
Sample constitutes digitized samples collection.
The mode of one-hot codings can make the characteristic of reset condition become sparse data, can preferably solve data
The problem of excavation classifies to attributive character data sample, and play the role of augmented features to a certain extent, wherein it is special
Sign data refer to feature and its corresponding value range.
For example, it is assumed that effective sample data include three features, respectively gender, area and browser, wherein gender
Characteristic value value range is：[male, female], regional characteristic value value range are：[Europe, US, Asia], browser
Characteristic value value range be：[Firefox, Chrome, Safari, Internet Explorer].
The characteristic value of each feature is encoded according to one-hot codings, obtained fundamental digital is encoded to：Male=
[1,0], female=[0,1], Europe=[1,0,0], US=[0,1,0], Asia=[0,0,1], Firefox=[1,0,
0,0], [0,1,0,0] Chrome=, Safari=[0,0,1,0], Internet Explorer]=[0,0,0,1].
If the characteristic value of some effective sample data is [male, US, Internet Explorer], then the effective sample
The corresponding digitized samples of data are：[1,0,0,1,0,0,0,0,1].
S5：Decision Tree algorithms are promoted to institute's digitized samples collection application gradient, generate the decision tree mould for including n decision tree
Type, wherein n is the positive integer more than 1.
Specifically, decision Tree algorithms are promoted to the digitized samples collection application gradient comprising digitized samples to be carried out to it
Modeling, predicts the feature of digitized samples by the decision-tree model of structure, and then obtain multiple branches, Mei Gefen
Branch includes the sample data of multiple same characteristic features.
Wherein, it is that one kind changes that gradient, which promotes decision tree (Gradient Boosting Decision Tree, GBDT) algorithm,
The decision Tree algorithms in generation, the algorithm are made of more decision trees, and the conclusion of all trees, which has added up, is used as final decision tree-model
Prediction result.
The decision tree that gradient is promoted in decision tree belongs to regression tree, and the node pair can be obtained in each node of these trees
The predicted value for the characteristic of division answered, for do not determine concrete numerical value characteristic of division, using the characteristic of division average value as
The predicted value of the characteristic of division.
S6：The feature that path using every decision tree in decision-tree model includes uses combination spy as assemblage characteristic
Sign carries out the model prediction of Logic Regression Models.
Specifically, the decision-tree model generated according to step S5, for each decision tree, the feature for including by different paths
Characteristic value carry out feature combination, obtain assemblage characteristic, and the value of the like combinations feature of different trees is added up, will be final
Characteristic value of the accumulated value as the assemblage characteristic, and return (Logistic using this feature value as two sorted logics
Regression, LR) independent variable in model, it is based on the two sorted logics regression model, calculates the independent variable default
Generation outline in event, and then predict whether the event is true according to preset probability threshold value.
It should be noted that the decision-tree model that step S5 is obtained is the feature coding model built, this feature coding
The output of model is the assemblage characteristic of each path of every decision tree, which can return directly as two sorted logics
Return the input feature vector of model, carry out model training and prediction, to eliminate the artificial process for finding assemblage characteristic, improves pair
The forecasting efficiency and predictablity rate of two sorted logic regression models.
For example, in a specific embodiment, referring to Fig. 2, Fig. 2 shows promote decision Tree algorithms using gradient to obtain
A specific regression tree, which is divided into sample data according to the age and is less than 30 years old and more than 30
The two nodes of year, then divide the two nodes by gender and educational background, obtain 5 nodes, respectively node 1, section
Point 2, node 3, node 4 and node 5, each node are an assemblage characteristic.Therefore, five are obtained according to the regression tree
Assemblage characteristic is respectively：Node 1 is corresponding " age is less than 30 and gender is female ", and node 2 is corresponding, and " age is less than 30, property
Not Wei man, and educational background be undergraduate course and its more than ", node 3 it is corresponding " age be less than 30, gender be man, and educational background be undergraduate course
Below ", node 4 it is corresponding " age be more than or equal to 30 and gender be female " and node 5 it is corresponding " age be more than or equal to 30
And gender is man ".
It should be noted that the embodiment of the present invention can be based on spark distributed computing frameworks to the process of data processing
It completes, to the structure of feature coding model, the advantage of spark Distributed Architecture to be made full use of, according to the configuration file automatically generated
In configuration information, initial sample data is carried out successively branch mailbox, one-hot coding and GBDT coding, efficiently quickly finish data
Effective data assemblage characteristic, implementation model prediction are extracted in processing.Since spark Distributed Architecture supports changing for big data quantity
Generation, therefore characteristic processing can be efficiently rapidly performed by when in face of huge initial sample data, modeling efficiency is improved, is shortened
The time of model publication from developing to reaching the standard grade.
In the corresponding embodiments of Fig. 1, initial sample data is being obtained according to configuration information, and according to preset configuration mould
After plate generates configuration file, the branch mailbox configuration information in configuration file is obtained first, according to the branch mailbox configuration information to initial sample
Notebook data carries out branch mailbox, and is handled initial sample data based on final branch mailbox result, obtains effective sample data, then right
Effective sample data carry out one-hot coding, and build digitized samples collection according to the result of one-hot coding, then to digitizing sample
This collection promotes decision Tree algorithms using gradient, generates the decision-tree model for including more decision trees, finally will be in decision-tree model
For the feature that the path of every decision tree includes as assemblage characteristic, the model that Logic Regression Models are carried out using the assemblage characteristic is pre-
It surveys, is built by carrying out branch mailbox, one-hot coding and decision tree successively to initial sample data, realize and initial sample data is carried out
Automatic accurate feature extraction so that obtained assemblage characteristic can accurately reflect the data characteristics of initial sample data, to
When carrying out model prediction according to the assemblage characteristic, the accuracy of model prediction can be effectively improved, meanwhile, by feature coding
The assemblage characteristic that model obtains can carry out model prediction directly as the input feature vector of two sorted logic regression models, to carry
High forecasting efficiency.
Next, on the basis of the corresponding embodiments of Fig. 1, step S2 refer to according to the configuration information, press
Branch mailbox after generating configuration file according to preset configuration template, and in the acquisition configuration file that step S3 is referred to matches confidence
Before breath, it can also be described to complete automatically updating for configuration file according to data.
Referring to Fig. 3, Fig. 3 show it is provided in an embodiment of the present invention according to data describe to complete configuration file it is automatic more
New specific implementation flow, details are as follows：
S81：Initial sample data is analyzed according to configuration file, obtains the data description of initial sample data.
In embodiments of the present invention, data description is the basic description of the default feature in initial sample data, this is basic
Description includes maximum value, minimum value, fractile or missing quantity of default feature etc., and data description can intuitively reflect default
The distribution situation of feature.
Wherein, maximum value and minimum value respectively refer in initial sample data the maximum occurrences of the default feature and minimum takes
Value, fractile are also referred to as quantile, refer to the numerical point that the distribution of a stochastic variable is divided into several equal portions, pass through
Fractile can analyze the variation tendency of the stochastic variable, and common fractile includes median, quartile, percentage
Digit etc., missing quantity refer to that the value of the default feature in initial sample data is the sample size of sky.
Default feature can specifically be predicted according to concrete model using needing be configured, for example, default feature can be with
It is " client actively initiates consulting total degree of insuring ", " client actively initiates consulting total degree of insuring in the first quarter " or " client
Nearest January actively initiates consulting number of insuring " etc..
Specifically, according to the parameter value for the configuration item being arranged in configuration file, initial sample data is extracted, and root
Initial sample data is handled according to default feature, obtains the characteristic value of the default feature, by default feature and its corresponding
Characteristic value constitutes the data description of the initial sample data.
For example, according to the configuration to Selection Model training set in configuration file, obtained from initial sample data corresponding
Training sample data, if default characterized by " client actively initiates consulting total degree of insuring " and " actively initiation in client's nearest January
Insure and seek advice from number ", then the basic description that the default feature is obtained from training sample data is as shown in Table 1：
Table one
S82：Data description is sent to user, so that user determines parameter to be adjusted according to data description.
Specifically, the data obtained in step S81 description is sent to user, user can be abundant according to data description
Understand the specific distribution situation for presetting feature, and further initial sample data analyze really according to specific distribution situation
Recognize, determines in configuration file need the configuration parameter being adjusted in time, and make rational adjust instruction.
Continue by taking the table one in step S81 as an example, when the data that user receives table one describe, if being retouched according to the data
It states and judges that the selection of training sample data is unreasonable, then confirm that the parameter value of the configuration item of Selection Model training set needs to carry out
Adjustment, and make the adjust instruction of the parameter value of the configuration item of the Selection Model training set reset.
S83：Receive the adjust instruction for treating adjusting parameter that user sends.
In embodiments of the present invention, adjust instruction is the instruction for being adjusted to the configuration parameter in configuration file,
The adjust instruction includes the targeted parameter value after configuration item to be adjusted and its corresponding adjustment.
Specifically, the adjust instruction that user sends out according to step S82 is received.
S84：According to adjust instruction, more new configuration file.
Specifically, the adjust instruction received according to step S83, obtained from the adjust instruction configuration item to be adjusted and
Its targeted parameter value, and use the parameter value of the configuration item in the targeted parameter value more new configuration file.
In the corresponding embodiments of Fig. 3, initial sample data is analyzed according to configuration file, obtains data description,
The distribution situation that can intuitively reflect the default feature of initial sample data, is conducive to user and is described according to data, determines in time
The configuration parameter adjusted is needed, rational adjust instruction is made, to according to adjust instruction, more new configuration file is received, obtain
To updated configuration file, and the structure of subsequent characteristics encoding model is carried out based on updated configuration file, realized pair
Model parameter carries out flexibly timely dynamic tuning, improves the accuracy and reasonability of configuration file, and then effectively improve spy
Levy the modeling efficiency and model accuracy rate of encoding model.
On the basis of the corresponding embodiments of Fig. 1, below by a specific embodiment come to being carried in step S4
And obtain configuration file in branch mailbox configuration information, according to the branch mailbox configuration information to initial sample data carry out branch mailbox, and
Initial sample data is handled based on final branch mailbox result, the concrete methods of realizing for obtaining effective sample data carries out in detail
Explanation.
Referring to Fig. 4, Fig. 4 shows the specific implementation flow of step S4 provided in an embodiment of the present invention, details are as follows：
S40：Null value filling information is obtained from configuration file, and according to null value filling information described in this to initial sample number
According to progress null value filling.
In embodiments of the present invention, it when the characteristic value of some feature of initial sample data is empty, needs to this feature
Value carries out null value filling, i.e., a preset value is arranged to it.The effect of null value filling is to allow the null value of feature to have and can solve
Analysis property.
Specifically, branch mailbox configuration information includes null value filling information, and control filling information, the control are obtained from configuration file
Filling information processed includes feature to be filled and its corresponding Filling power.According to the feature to be filled got, to initial sample number
According to being traversed, if the characteristic value of the feature to be filled of initial sample data is sky, the corresponding filling of feature to be filled is used
Value is filled.S41：Branch mailbox feature is obtained from configuration file.
In embodiments of the present invention, branch mailbox configuration information further includes branch mailbox feature, and branch mailbox is characterized as needing to carry out branch mailbox
Characteristic attribute, such as age.Branch mailbox feature specifically can be flexibly arranged by user according to modeling requirement or using needs.
S42：It is determining from initial sample data to wait for that the nominal variable of branch mailbox is corresponding with the nominal variable according to branch mailbox feature
M characteristic value, wherein m is positive integer more than 1.
In embodiments of the present invention, characteristic attribute includes continuous variable and nominal variable two types, and continuous variable refers to
Its characteristic value in certain section can arbitrary value variable, characteristic value be continuously, any two characteristic value it
Between can have unit with limitless defense right and can sort, such as distance；Nominal variable, which refers to its characteristic value, to be enumerated,
But without unit also without sequence, such as gender.
If branch mailbox feature belongs to continuous variable, discretization is carried out to continuous variable first, and to continuous after discretization
Variable extracts corresponding nominal variable and its corresponding m characteristic value, i.e., converts continuous variable to nominal variable；If root case is special
Sign belongs to nominal variable, then determines that this waits for the corresponding m characteristic value of the nominal variable of branch mailbox directly from sample data.
For example, it is assumed that branch mailbox is characterized as educational background, i.e. nominal variable is educational background, then the name is determined from initial sample data
The value range of variable is：Primary school, junior middle school, senior middle school, undergraduate course, postgraduate or more, totally 5 characteristic values.
S43：The initial value that m characteristic value is stored into preset characteristic value collection, and branch mailbox wheel number k is arranged is 0, with
And the 0th wheel branch mailbox branch mailbox result be sky, wherein k be more than or equal to 0, and be less than or equal to m-1.
In embodiments of the present invention, the m characteristic value that step S42 is got is saved in preset characteristic value collection,
And initialize branch mailbox wheel number k, be arranged k initial value be 0, while give tacit consent to k equal to 0 when, the 0th wheel branch mailbox branch mailbox result be sky,
The value range of branch mailbox wheel number k is more than or equal to 0 and to be less than or equal to m-1.
It should be noted that preset characteristic value collection is used to store the characteristic value of nominal variable, for subsequently according to feature
Value carries out branch mailbox and prepares.
For example, it is assumed that three characteristic values of nominal variable are：1560,2240 and 3200, then its whole is deposited into default
Characteristic value collection among, due to not carrying out branch mailbox operation also, i.e., branch mailbox wheel number is 0, therefore there is no branch mailbox as a result, the 0th wheel point
The branch mailbox result of case is sky.
S44：It is test split point with this feature value, in kth wheel branch mailbox for each characteristic value in characteristic value collection
Nominal variable is divided into k+2 casees on the basis of branch mailbox result, this feature is calculated and is worth corresponding coupling index value, obtains m-k pass
Join index value.
In embodiments of the present invention, it using the characteristic value in characteristic value collection as test split point, is divided by the test
Point carries out branch mailbox to nominal variable, according to the m characteristic value got, m test split point is obtained, to each test split point
A branch mailbox operation is carried out, to execute m branch mailbox operation.
Specifically, as k=0, i.e. the 0th roller box, expression does not carry out branch mailbox；When carrying out the 1st wheel branch mailbox operation, exist at this time
Do not carry out to wait for that the nominal variable of branch mailbox is divided into 2 casees by 1 split point on the basis of branch mailbox, i.e., in k=0, the 0th wheel not
Nominal variable is divided into 2 casees on the basis of progress branch mailbox, is k+2 casees；When carrying out the 2nd wheel branch mailbox, branch mailbox is taken turns the 1st at this time
As a result on the basis of 2 casees, by 1 split point to wherein including to be divided into 2 casees 1 case of the split point, name is become in total at this time
Amount is divided into 3 casees, i.e., in k=1, nominal variable is divided into 3 casees on the basis of the 1st wheel branch mailbox result 2 casees, is k+2 casees.With
This analogizes, and obtains to wait for that the nominal variable of branch mailbox is divided on the basis of the be k+1 wheel branch mailbox results is in kth wheel branch mailbox result
K+2 casees.
Also, during each round branch mailbox operates, the corresponding coupling index value of each test split point is calculated,
In, the number of coupling index value is the characteristic value number in current signature value set, the i.e. difference of m and branch mailbox wheel number k.
It should be noted that during each round branch mailbox operates, coupling index value can be the value of information
(information value, IV), Geordie variance index value or Pearson came chi-square statistics amount etc..Wherein, IV values are a kind of measurements
The coefficient of the predictive ability size of independent variable, Geordie variance index value refer to that impurity level drops after sample set is divided by particular community
Low ratio, Pearson came chi-square statistics amount are used to weigh the correlation between two nominal variables.
S45：Using the corresponding characteristic value of maximum value in m-k coupling index value as Target Splitting point, in kth wheel branch mailbox
Branch mailbox result on the basis of nominal variable is divided into k+2 casees, take turns the branch mailbox of branch mailbox as kth+1 as a result, and by this feature value
It is removed from characteristic value collection.
In embodiments of the present invention, in m-k coupling index value being calculated from step S44, maximum is chosen
The corresponding characteristic value of coupling index value carries out branch mailbox, each round branch mailbox all bases as Target Splitting point, according to the Target Splitting point
On the basis of last round of branch mailbox result, the nominal variable comprising the Target Splitting point is divided by 2 casees according to Target Splitting point, therefore
Kth+1 takes turns the branch mailbox of branch mailbox the result is that adding 2 on the basis of the branch mailbox result based on kth wheel branch mailbox, i.e. k+2 casees are the wheel point of kth+1
The result of case.
It, will after having executed branch mailbox operation meanwhile as the corresponding characteristic value of most relevance index value of Target Splitting point
It is removed from characteristic value collection.
S46：If k+2 reaches preset case number threshold value, stop branch mailbox, and kth+1 is taken turns to the branch mailbox result determination of branch mailbox
For final branch mailbox as a result, otherwise, carrying out return to step S44 after adding 1 operation to k and continuing to execute.
Specifically, it is k+2 casees that can obtain kth+1 to take turns the result of branch mailbox according to step S45, if branch mailbox result k+2 reaches default
Case number threshold value, then do not continue to carry out branch mailbox, and using the k+2 casees as final branch mailbox result；If branch mailbox result k+2 casees are not
Reach preset case number threshold value, then return to step S44 continues new round branch mailbox after adding 1 to k.
Further, during branch mailbox, pass can also be worked as using coupling index value as the Rule of judgment for stopping branch mailbox
Join index value enhancing rate be less than preset enhancing rate threshold value when, stop branch mailbox, otherwise, to k carry out plus 1 operation after return to step
S44 continues new round branch mailbox.
Wherein, the enhancing rate of coupling index value can specifically be calculated according to following formula, and details are as follows：
V=(Xp-Xp-1)/Xp
Wherein, v is the enhancing rate of coupling index value, XpFor the determining corresponding pass of Target Splitting point of pth wheel branch mailbox operation
Join index value, p ∈ [1, m].
It should be noted that in sample data initial in face of magnanimity, the branch mailbox process of the embodiment of the present invention can be based on
Spark distributed computing frameworks carry out, and by spark Distributed Parallel Computings, can improve computational efficiency, meanwhile, to big
Data volume at the same have multiple nominal variables need carry out branch mailbox when, can effectively improve branch mailbox efficiency.
S47：According to final branch mailbox as a result, determining the branch mailbox characteristic value of the branch mailbox feature of initial sample data, obtain effectively
Sample data.
Specifically, the final branch mailbox result that step S46 is obtained includes the branch mailbox number of branch mailbox feature and the branch mailbox spy per case
Value indicative determines that this is initial based on the final branch mailbox as a result, according to the initial characteristic values of the branch mailbox feature of each initial sample data
The corresponding branch mailbox characteristic value of characteristic value obtains the effective sample data for including the branch mailbox characteristic value.
By taking branch mailbox is characterized as the age as an example, if finally branch mailbox result is [10,35), [35,45), [45,60) and [60,80]
Totally four case, i.e. branch mailbox number are 4, the branch mailbox characteristic value per case be respectively [10,35), [35,45), [45,60) and [60,80].It is false
If the initial characteristic values at the age of certain initial sample data are 50, then 50 range for belonging to [60,80], the i.e. initial characteristic values pair
The branch mailbox characteristic value answered is [60,80], therefore point for including in the obtained corresponding effective sample data of the initial sample data
Case characteristic value is [60,80].
In the corresponding embodiments of Fig. 4, null value filling is carried out to initial sample data first so that initial sample data
The characteristic value of each feature all has resolvability, and branch mailbox feature is then obtained from configuration file, according to the branch mailbox feature from
It is determined in initial sample data and waits for the nominal variable of branch mailbox characteristic value corresponding with the nominal variable, and characteristic value is stored to pre-
If characteristic value collection in, during each round branch mailbox, using each characteristic value in characteristic value collection as test split point
Nominal variable is divided into two casees, and calculates the corresponding coupling index value of each characteristic value, maximum value is chosen from coupling index value
Corresponding characteristic value executes branch mailbox operation as Target Splitting point, stops dividing if branch mailbox result reaches preset case number threshold value
Otherwise case continues to execute branch mailbox operation, realize and carry out automatic branch mailbox to nominal variable based on coupling index value so that in maximum
While degree preserves initial sample data information, feature extraction is rapidly and accurately carried out, to reduce manual intervention and take,
The branch mailbox efficiency for improving branch mailbox operation, so as to rapid build feature coding model.
On the basis of the corresponding embodiments of Fig. 1, one-hot coding is carried out to effective sample data what step S5 was referred to,
And according to the result of one-hot coding build digitized samples collection after, and step S6 refer to digitize sample set application
Before gradient promotes decision Tree algorithms, across variable coding can also be carried out to the digitized samples in digitlization sample set.
Referring to Fig. 5, Fig. 5 shows that the digitized samples in the sample set provided in an embodiment of the present invention to digitlization carry out
The specific implementation flow of across variable coding, details are as follows：
S91：According to the cross-over configuration information in configuration file, the digitized samples in digitlization sample set are intersected
Variable encodes, and obtains the cross feature value of the cross feature of each digitized samples.
In embodiments of the present invention, cross-over configuration information includes to wait for the feature of combined crosswise.
Specifically, according to the cross-over configuration information in configuration file, across variable coding is carried out to digitized samples, is obtained
The detailed process of the cross feature value of the cross feature of each digitized samples includes step a) to step c), and details are as follows：
A) the N number of foundation characteristic for waiting for combined crosswise, and the value range of each foundation characteristic are obtained.
Specifically, the cross-over configuration information in configuration file is read, obtains the N number of foundation characteristic for waiting for combined crosswise, and from
Digitized samples concentrate the value range for determining each foundation characteristic, finally obtain each foundation characteristic and corresponding all spies
Value indicative, it is to be understood that each characteristic value is to be encoded according to the fundamental digital that one-hot coding mode obtains.
It should be noted that N is positive integer, minimum value 2 at least can carry out intersection group to 2 foundation characteristics
It closes.
B) each characteristic value of each foundation characteristic corresponding fundamental digital coding is traversed, from each foundation characteristic
It is middle to choose a fundamental digital coding progress combined crosswise calculating respectively, obtain each combination of the characteristic value of N number of foundation characteristic
The corresponding combination digital coding of mode.
Specifically, the corresponding fundamental digital coding of each characteristic value of each foundation characteristic obtained in step a) is carried out
It traverses one by one, the characteristic value of each foundation characteristic and the characteristic value of other foundation characteristics is combined, that is, choose each
The fundamental digital coding of foundation characteristic is encoded with other fundamental digitals different from the foundation characteristic belonging to itself to be handed over
Fork combination, obtains the corresponding combination digital coding of each combination, i.e., feature combination at this time is compiled by a string of combination numbers
Code indicates.
In embodiments of the present invention, by be based on spark distributed computing frameworks can make any one foundation characteristic into
Row combined crosswise, that is, by a fundamental digital of each foundation characteristic coding with it is other special different from the basis belonging to itself
The fundamental digital coding of sign carries out combined crosswise, effectively increases combined crosswise computational efficiency.
For example, it is assumed that foundation characteristic is gender and area, wherein the value range of gender is [male, female], area
Value range be [Europe, US, Asia].The combined crosswise difference that the characteristic value of each foundation characteristic is intersected
For：[male, Europe], [male, US], [male, Asia], [female, Europe], [female, US] and [female,
Asia] totally 6 combinations.
By taking combined crosswise [male, US] as an example, by male corresponding fundamental digital codings [1,0] each with US pairs
Each in the fundamental digital coding [0,1,0] answered carries out combined crosswise calculating, obtains the corresponding basic number of the combined crosswise
Word coding is respectively [1,0], [1,1], [1,0], [0,0], [0,1] and [0,0] six combinations, by two volumes in each combination
Code, which is multiplied, respectively obtains 0,1,0,0,0 and 0, and final combination obtains the corresponding combination digital coding of combination of the combined crosswise
For [0,1,0,0,0,0], similarly, the combination of other five combined crosswises and its corresponding combination digital coding are respectively
[male, Europe] corresponding [1,0,0,0,0,0], [male, Asia] corresponding [0,0,1,0,0,0], [female, Europe] are right
Answer [0,0,0,1,0,0], [female, US] corresponding [0,0,0,0,1,0] and [female, Asia] corresponding [0,0,0,0,0,1].
C) according to each combination and its corresponding combination digital coding, the cross feature of each digitized samples is determined
Cross feature value.
Specifically, the foundation characteristic in digitized samples and the characteristic value corresponding to it are read, is obtained according to step c)
Each combination and its corresponding combination digital coding, determine the combination that digitized samples are matched to, and by the combination
The cross feature value of cross feature of the corresponding combination digital coding of mode as the digitized samples.
For example, continue to quote the example of step c), according to six kinds of combinations [male, Europe], [male, US],
[male, Asia], [female, Europe], [female, US] and [female, Asia] and the number of combinations corresponding to them
Word encode [1,0,0,0,0,0], [0,1,0,0,0,0], [0,0,1,0,0,0], [0,0,0,1,0,0], [0,0,0,0,1,0] and
[0,0,0,0,0,1], when the gender of some digitized samples is female, area is Europe, by combination
The friendship of cross feature of [female, Europe] the corresponding combination digital coding [0,0,0,1,0,0] as the digitized samples
Pitch characteristic value.
S92：Using the digitized samples comprising cross feature, digitized samples collection is updated.
Specifically, according to the across variable of step S91 coding as a result, increasing cross feature and its right to digitized samples
The cross feature value answered obtains updated digitized samples collection.
It should be noted that with the increase of N, the calculation amount of combined crosswise calculating is carried out in across variable cataloged procedure
It increases rapidly, in embodiments of the present invention, can be realized to any number of basis by using spark distributed computing frameworks
Across variable coding between feature, effectively increases combined crosswise computational efficiency.
In the corresponding embodiments of Fig. 5, by carrying out across variable coding to the digitized samples in digitlization sample set,
It chooses a fundamental digital coding respectively from each foundation characteristic and is combined calculating, obtain the characteristic value of N number of foundation characteristic
The corresponding combination digital coding of each combination, and then determine according to the combination digital coding friendship of each digitized samples
The cross feature value for pitching feature, realizes the expression of the nonlinear characteristic to initial sample data so that is encoded in construction feature
The non-linear relation between feature can be added when model, improve the accuracy of model construction.
On the basis of the corresponding embodiments of Fig. 1, step S6 is referred to below by a specific embodiment pair
Digitized samples collection application gradient promotes decision Tree algorithms, generates the specific implementation side of the decision-tree model comprising n decision tree
Method is described in detail.
Referring to Fig. 6, Fig. 6 shows the specific implementation flow of step S6 provided in an embodiment of the present invention, details are as follows：
S61：Based on Spark distributed computing frameworks, the digitized samples in digitlization sample set are returned using classification
Tree algorithm generates original decision tree.
In embodiments of the present invention, post-class processing (Classification And Regression Tree, CART)
Algorithm is also known as least square regression tree, and CART algorithms have the possibility as leaf node in view of each node, to each
Node all distributes classification.Distribution class method for distinguishing can also be referred to occurring most classifications in present node and be worked as prosthomere
Point classification error or other more complicated methods, be using a kind of based on two points of recursive subdivisions by the way of, the algorithm is always
Current sample set is divided into two sub- sample sets so that only there are two branches for each leafy node of the decision tree of generation.
Therefore the decision tree that CART algorithms generate is binary tree simple for structure, the value that CART algorithms are suitable for sample characteristics be or
Non- scene.
Wherein, each node of post-class processing can obtain a predicted value, and by taking the age as an example, which, which is equal to, belongs to
In the average value at owner's age of this node.In branch, the corresponding digitlization variate-value of each feature of exhaustion is looked for best
Cut-point, but it is no longer maximum entropy to weigh best standard, but minimizes square error as cutting error, that is, by
Predict that the number of error is more, wrong more goes against accepted conventions, and cutting error is bigger, is used as cutting error energy by minimizing square error
Enough find most reliable branch foundation.If the age of people is not unique on final leaf node, with proprietary flat on the node
Prediction age of the equal age as the leaf node.
Specifically, Spark distributed computing frameworks are based on, the digitized samples in digitlization sample set are calculated using CART
Method generates original decision tree.
S62：Original decision tree is put into decision-tree model, and using the original decision tree as current decision tree.
Specifically, after the generation of original decision tree, which is put into preset decision-tree model
In, decision-tree model can include more decision trees.
Meanwhile using original decision tree as current decision tree, to carry out next round decision tree fitting.
S63：Based on digitized samples collection, the residual vector of current decision tree is calculated.
Specifically, in gradient promotes decision Tree algorithms, Weak Classifier is constructed using the practice of gradient promotion, each
When iteration, loss of the digitized samples on each characteristic of division in current decision tree is calculated by using loss function
Value, and then new decision tree is generated to be fitted using the penalty values as the predicted value of next tree, the penalty values be residual error to
The absolute value of amount.Wherein, characteristic of division refers to that digitized samples carry out every time when generating original decision tree using CART algorithms
Characteristic attribute used in cutting.
Wherein, loss function includes but not limited to：0-1 loss functions (0-1Loss Function), quadratic loss function
(Quadratic Loss Function), absolute error loss function (Absolute Loss Function) and logarithm lose letter
Number (Logarithmic Loss Function) etc..
Preferably, loss function used in the embodiment of the present invention is logarithm loss function, which uses greatly
The method of possibility predication.
S64：Residual vector according to current decision tree is fitted new decision tree, and the new decision tree is put into decision tree
In model.
Specifically, using the corresponding residual vector of the characteristic of division of digitized samples as new decision tree in classification spy
Predicted value in sign carries out the fitting of new decision tree according to step S61 so that new decision tree makees into one current decision tree
What is walked is perfect, improves accuracy rate of the decision-tree model for the feature description of digitized samples.
S65：If the sum of decision tree is less than predetermined threshold value in decision-tree model, using new decision tree as current decision
Tree, return to step S63 are continued to execute.
Specifically, when new decision tree is added to decision in model, the decision tree sum in statistical decision tree-model,
If decision tree sum is less than default decision tree amount threshold, then it is assumed that the unstructured completion of decision-tree model, return to step S63 continue
It executes, continues to be fitted new decision tree by calculating the residual vector of current decision tree.
Wherein, presetting decision tree amount threshold can be configured in configuration file.
S66：If the sum of decision tree reaches predetermined threshold value in decision-tree model, stop being fitted new decision tree.
Specifically, it when new decision tree is added to decision in model, counts on decision tree sum and reaches default decision
Set amount threshold, then it is assumed that decision-tree model structure is completed, and the fitting to new decision tree, and the decision that will be obtained at this time are stopped
Tree-model is as final decision-tree model.
In the corresponding embodiments of Fig. 6, Spark distributed computing frameworks are based on, original decision is generated by CART algorithms
Tree, original decision tree is put into decision-tree model, and using the original decision tree as current decision tree, and then calculates current determine
The residual vector of plan tree is fitted new decision tree according to the residual vector of current decision tree, and the new decision tree is put into certainly
In plan tree-model, so cycle is fitted new decision tree, until the sum of decision tree in decision-tree model reaches predetermined threshold value, then
Stop being fitted new decision tree, whole process promotes decision Tree algorithms using gradient so that each new decision tree is all pair
The fitting of current decision tree has gradually reduced the error of decision-tree model, and the characteristic of division prediction for improving digitized samples is accurate
True rate improves the assemblage characteristic to digitized samples by the way of obtaining assemblage characteristic using gradient promotion decision Tree algorithms
The accuracy rate of prediction also improves the efficiency that assemblage characteristic obtains, meanwhile, using Spark distributed computing frameworks, improve logarithm
The data processing speed of word sample, to effectively improve the structure efficiency of decision-tree model.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
Embodiment 2
Corresponding to the method for the data processing in embodiment 1, Fig. 7 shows the side of the data processing provided with embodiment 1
The device of the one-to-one data processing of method illustrates only and the relevant part of the embodiment of the present invention for convenience of description.
As shown in fig. 7, the device of the data processing includes：Data acquisition module 71, file generating module 72, branch mailbox module
73, digital module 74, decision tree structure module 75 and model prediction module 76.Detailed description are as follows for each function module：
Data acquisition module 71 obtains initial sample data for obtaining configuration information, and based on the configuration information；
File generating module 72, for according to configuration information, configuration file to be generated according to preset configuration template；
Branch mailbox module 73, for obtaining the branch mailbox configuration information in configuration file, according to the branch mailbox configuration information to initial
Sample data carries out branch mailbox, and is handled initial sample data based on final branch mailbox result, obtains effective sample data,
In, effective sample data include the branch mailbox characteristic value of branch mailbox feature；
Digital module 74 for carrying out one-hot coding to effective sample data, and is built according to the result of one-hot coding
Digitized samples collection；
Decision tree builds module 75, and for promoting decision Tree algorithms to digitlization sample set application gradient, it includes n to generate
The decision-tree model of decision tree, wherein n is the positive integer more than 1；
Model prediction module 76, the feature for including using the path of every decision tree in decision-tree model are special as combination
Sign carries out the model prediction of Logic Regression Models using the assemblage characteristic.
Further, the device of the data processing further includes：
Data describing module 771 obtains initial sample number for being analyzed initial sample data according to configuration file
According to data description；
Sending module 772, for data description to be sent to user, so that user is to be adjusted according to data description determination
Parameter；
Receiving module 773, the adjust instruction for treating adjusting parameter for receiving user's transmission；
File update module 774, for according to adjust instruction, more new configuration file.
Further, branch mailbox configuration information includes null value filling information and branch mailbox feature, and branch mailbox module 73 includes：
Null value fills submodule 730, for obtaining null value filling information from configuration file, and is filled and is believed according to the null value
Breath carries out null value filling to initial sample data；
Feature acquisition submodule 731, for obtaining branch mailbox feature from configuration file；
Variable determination sub-module 732 waits for that the name of branch mailbox becomes for according to branch mailbox feature, being determined from initial sample data
Amount m characteristic value corresponding with the nominal variable, wherein m is the positive integer more than 1；
For storing m characteristic value into preset characteristic value collection, and branch mailbox wheel is arranged in initialization submodule 733
The initial value of number k is that the branch mailbox result of the 0 and the 0th wheel branch mailbox is empty, wherein k is more than or equal to 0, and is less than or equal to m-1；
Branch mailbox tests submodule 734, for being test point with this feature value for each characteristic value in characteristic value collection
Nominal variable is divided into k+2 casees by knick point on the basis of the branch mailbox result of kth wheel branch mailbox, is calculated this feature and is worth corresponding association
Index value obtains m-k coupling index value；
Branch mailbox determination sub-module 735, for using the corresponding characteristic value of maximum value in m-k coupling index value as target
Nominal variable is divided into k+2 casees by split point on the basis of the branch mailbox result of kth wheel branch mailbox, and the branch mailbox of branch mailbox is taken turns as kth+1
As a result, and this feature value is removed from characteristic value collection；
Branch mailbox submodule 736 is recycled, if reaching preset case number threshold value for k+2, stops branch mailbox, and kth+1 is taken turns
The branch mailbox result of branch mailbox is determined as final branch mailbox as a result, otherwise, to k carry out plus 1 operation after return branch mailbox test submodule 734 after
It is continuous to execute；
As a result determination sub-module 737 are used for according to final branch mailbox as a result, determining point of the branch mailbox feature of initial sample data
Case characteristic value obtains effective sample data.
Further, the device of the data processing further includes：
Intersect coding module 781, is used for according to the cross-over configuration information in configuration file, to the number in digitlization sample set
Word sample carries out across variable coding, obtains the cross feature value of the cross feature of each digitized samples；
Data update module 782 updates digitized samples collection for using the digitized samples comprising cross feature.
Further, decision tree structure module 75 includes：
Raw element tree builds submodule 751, for being calculated using post-class processing the digitized samples in digitlization sample set
Method generates original decision tree；
Decision tree updates submodule 752, for original decision tree to be put into decision-tree model, and by the original decision tree
As current decision tree；
Residual computations submodule 753 calculates the residual vector of current decision tree for being based on digitized samples collection；
It is fitted submodule 754, new decision tree is fitted for the residual vector according to current decision tree, and this new is determined
Plan tree is put into decision-tree model；
Cycle fitting submodule 755 will be new if the sum for decision tree in decision-tree model is less than predetermined threshold value
Decision tree returns to residual computations submodule 753 and continues to execute as current decision tree；
Decision tree output sub-module 757 stops if the sum for decision tree in decision-tree model reaches predetermined threshold value
It is fitted new decision tree.
Each module realizes the process of respective function in a kind of device of data processing provided in this embodiment, specifically refers to
The description of previous embodiment 1, details are not described herein again.
Embodiment 3
The present embodiment provides a computer readable storage medium, computer journey is stored on the computer readable storage medium
Sequence, the method that data processing in embodiment 1 is realized when which is executed by processor, alternatively, the computer program quilt
The function of each module/unit in the device of data processing in embodiment 2 is realized when processor executes.To avoid repeating, here not
It repeats again.
It is to be appreciated that the computer readable storage medium may include：The computer program code can be carried
Any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic disc, CD, computer storage, read-only memory
(Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electric carrier signal and
Telecommunication signal etc..
Embodiment 4
Fig. 8 is the schematic diagram for the terminal device that one embodiment of the invention provides.As shown in figure 8, the terminal of the embodiment is set
Standby 80 include：Processor 81, memory 82 and it is stored in the computer journey that can be run in memory 82 and on processor 81
Sequence 83, for example, data processing program.Processor 81 executes the method for realizing above-mentioned each data processing when computer program 83
Step in embodiment, such as step S1 shown in FIG. 1 to step S6.Alternatively, reality when processor 81 executes computer program 83
The function of each module/unit in existing above-mentioned each device embodiment, such as module 71 shown in Fig. 7 is to the function of module 76.
Illustratively, computer program 83 can be divided into one or more module/units, one or more mould
Block/unit is stored in memory 82, and is executed by processor 81, to complete the present invention.One or more module/units can
To be the series of computation machine program instruction section that can complete specific function, the instruction segment is for describing computer program 83 at end
Implementation procedure in end equipment 80.For example, computer program 83 can be divided into data acquisition module, file generating module,
Branch mailbox module, digital module, decision tree structure module and model prediction module, each module concrete function is as described in Example 2,
To avoid repeating, do not repeat one by one herein.
Terminal device 80 can be the computing devices such as desktop PC, notebook, palm PC and cloud server.Eventually
End equipment 80 may include, but be not limited only to, processor 81, memory 82.It will be understood by those skilled in the art that Fig. 8 is only
The example of terminal device 80 does not constitute the restriction to terminal device 80, may include components more more or fewer than diagram, or
Person combines certain components or different components, such as terminal device 80 can also be set including input-output equipment, network insertion
Standby, bus etc..
Alleged processor 81 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), application-specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor can also be any conventional processor
Deng.
Memory 82 can be the internal storage unit of terminal device 80, such as the hard disk or memory of terminal device 80.It deposits
Reservoir 82 can also be the plug-in type hard disk being equipped on the External memory equipment of terminal device 80, such as terminal device 80, intelligence
Storage card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash card (Flash Card)
Deng.Further, memory 82 can also both include terminal device 80 internal storage unit and also including External memory equipment.It deposits
Reservoir 82 is used to store other programs and data needed for computer program and terminal device 80.Memory 82 can be also used for
Temporarily store the data that has exported or will export.
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each work(
Can unit, module division progress for example, in practical application, can be as needed and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device are divided into different functional units or module, more than completion
The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although with reference to aforementioned reality
Applying example, invention is explained in detail, it will be understood by those of ordinary skill in the art that：It still can be to aforementioned each
Technical solution recorded in embodiment is modified or equivalent replacement of some of the technical features；And these are changed
Or replace, the spirit and scope for various embodiments of the present invention technical solution that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (10)
1. a kind of method of data processing, which is characterized in that the method includes：
Configuration information is obtained, and initial sample data is obtained based on the configuration information；
According to the configuration information, configuration file is generated according to preset configuration template；
Obtain the branch mailbox configuration information in the configuration file, according to the branch mailbox configuration information to the initial sample data into
Row branch mailbox, and the initial sample data is handled based on final branch mailbox result, obtain effective sample data, wherein institute
State the branch mailbox characteristic value that effective sample data include branch mailbox feature；
One-hot coding is carried out to the effective sample data, and digitized samples collection is built according to the result of the one-hot coding；
Decision Tree algorithms are promoted to the digitized samples collection application gradient, generate the decision-tree model for including n decision tree,
In, n is the positive integer more than 1；
The feature that path using every decision tree in the decision-tree model includes uses the assemblage characteristic as assemblage characteristic
Carry out the model prediction of Logic Regression Models.
2. the method for data processing as described in claim 1, which is characterized in that it is described according to the configuration information, according to pre-
If configuration template generate configuration file after, and before the branch mailbox configuration information obtained in the configuration file, institute
The method of stating further includes：
Initial sample data is analyzed according to the configuration file, obtains the data description of the initial sample data；
Data description is sent to user, so that the user determines parameter to be adjusted according to data description；
Receive the adjust instruction to the parameter to be adjusted that the user sends；
According to the adjust instruction, the configuration file is updated.
3. the method for data processing as described in claim 1, which is characterized in that the branch mailbox configuration information includes null value filling
Information and branch mailbox feature, the branch mailbox configuration information obtained in the configuration file, according to the branch mailbox configuration information to institute
It states initial sample data and carries out branch mailbox, and the initial sample data is handled based on final branch mailbox result, obtain effectively
Sample data includes：
The null value filling information is obtained from the configuration file, and according to the null value filling information to the initial sample
Data carry out null value filling；
The branch mailbox feature is obtained from the configuration file；
It is determining from the initial sample data to wait for the nominal variable of branch mailbox and the nominal variable pair according to the branch mailbox feature
The m characteristic value answered, wherein m is the positive integer more than 1；
The initial value that the m characteristic values are stored into preset characteristic value collection, and branch mailbox wheel number k is arranged is 0, Yi Ji
The branch mailbox result of 0 wheel branch mailbox is sky, wherein k is more than or equal to 0, and is less than or equal to m-1；
It is test split point with this feature value, in the branch mailbox of kth wheel branch mailbox for each characteristic value in the characteristic value collection
As a result the nominal variable is divided into k+2 casees on the basis of, calculates the corresponding coupling index value of the characteristic value, obtains m-k
The coupling index value；
Using the corresponding characteristic value of maximum value in the m-k coupling index values as Target Splitting point, in point of kth wheel branch mailbox
The nominal variable is divided into k+2 casees on the basis of case result, takes turns the branch mailbox of branch mailbox as kth+1 as a result, and by this feature value
It is removed from the characteristic value collection；
If k+2 reaches preset case number threshold value, stop branch mailbox, and the branch mailbox result of the kth+1 wheel branch mailbox is determined as most
Otherwise whole branch mailbox is as a result, carry out k to return to each characteristic value in the characteristic value collection after adding 1 operation, with this
Characteristic value is that the nominal variable is divided into k+2 casees by test split point on the basis of the branch mailbox result of kth wheel branch mailbox, is calculated
This feature is worth corresponding coupling index value, and the step of obtaining the m-k coupling index values continues to execute；
According to the final branch mailbox as a result, determining the branch mailbox characteristic value of the branch mailbox feature of the initial sample data, obtain
Effective sample data.
4. the method for data processing as described in claim 1, which is characterized in that described to be carried out solely to the effective sample data
Heat coding, and according to after the result of one-hot coding structure digitized samples collection and described to the digitized samples
Collection promotes decision Tree algorithms using gradient, and before generating the decision-tree model comprising n decision tree, the method further includes：
According to the cross-over configuration information in the configuration file, the digitized samples concentrated to the digitized samples are intersected
Variable encodes, and obtains the cross feature value of the cross feature of each digitized samples；
Using the digitized samples comprising the cross feature, the digitized samples collection is updated.
5. the method for data processing as described in claim 1, which is characterized in that described to digitized samples collection application ladder
Degree promotes decision Tree algorithms, generates the decision-tree model comprising n decision tree and includes：
Post-class processing algorithm is used to the digitized samples that the digitized samples are concentrated, generates original decision tree；
The original decision tree is put into decision-tree model, and using the original decision tree as current decision tree；
Based on the digitized samples collection, the residual vector of the current decision tree is calculated；
It is fitted new decision tree according to the residual vector, and the new decision tree is put into the decision-tree model；
If the sum of decision tree is less than predetermined threshold value in the decision-tree model, using the new decision tree as described current
The step of decision tree, return is described based on the digitized samples collection, the residual vector for calculating the current decision tree continues to hold
Row；
If the sum of decision tree reaches predetermined threshold value in the decision-tree model, stop being fitted new decision tree.
6. a kind of device of data processing, which is characterized in that described device includes：
Data acquisition module obtains initial sample data for obtaining configuration information, and based on the configuration information；
File generating module, for according to the configuration information, configuration file to be generated according to preset configuration template；
Branch mailbox module, for obtaining the branch mailbox configuration information in the configuration file, according to the branch mailbox configuration information to described
Initial sample data carries out branch mailbox, and is handled the initial sample data based on final branch mailbox result, obtains effective sample
Notebook data, wherein the effective sample data include the branch mailbox characteristic value of branch mailbox feature；
Digital module, for carrying out one-hot coding to the effective sample data, and according to the result structure of the one-hot coding
Build digitized samples collection；
Decision tree builds module, for promoting decision Tree algorithms to the digitized samples collection application gradient, generates comprising n certainly
The decision-tree model of plan tree, wherein n is the positive integer more than 1；
Model prediction module, the feature for including using the path of every decision tree in the decision-tree model are special as combination
Sign carries out the model prediction of Logic Regression Models using the assemblage characteristic.
7. the device of data processing as claimed in claim 6, which is characterized in that described device further includes：
Data describing module obtains the initial sample for being analyzed initial sample data according to the configuration file
The data of data describe；
Sending module, for data description to be sent to user, so that the user waits for according to data description determination
Adjusting parameter；
Receiving module, the adjust instruction to the parameter to be adjusted sent for receiving the user；
File update module, for according to the adjust instruction, updating the configuration file.
8. the device of data processing as claimed in claim 6, which is characterized in that described device further includes：
Intersect coding module, for according to the cross-over configuration information in the configuration file, being concentrated to the digitized samples
Digitized samples carry out across variable coding, obtain the cross feature value of the cross feature of each digitized samples；
Data update module updates the digitized samples for using the digitized samples comprising the cross feature
Collection.
9. a kind of terminal device, including memory, processor and it is stored in the memory and can be on the processor
The computer program of operation, which is characterized in that the processor realizes such as claim 1 to 5 when executing the computer program
The step of method of any one data processing.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, feature to exist
In the step of realization method of data processing as described in any one of claim 1 to 5 when the computer program is executed by processor
Suddenly.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810309823.0A CN108764273B (en) | 2018-04-09 | 2018-04-09 | Data processing method, device, terminal equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810309823.0A CN108764273B (en) | 2018-04-09 | 2018-04-09 | Data processing method, device, terminal equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108764273A true CN108764273A (en) | 2018-11-06 |
| CN108764273B CN108764273B (en) | 2023-12-05 |
Family
ID=63981389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810309823.0A Active CN108764273B (en) | 2018-04-09 | 2018-04-09 | Data processing method, device, terminal equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108764273B (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109919783A (en) * | 2019-01-31 | 2019-06-21 | 德联易控科技(北京)有限公司 | Risk Identification Method, device, equipment and the storage medium of vehicle insurance Claims Resolution case |
| CN109978056A (en) * | 2019-03-26 | 2019-07-05 | 广东工业大学 | A kind of Metro Passenger classification method based on machine learning |
| CN110070270A (en) * | 2019-03-29 | 2019-07-30 | 广东民航机场建设有限公司 | Transmission equipment control method and device for connecting bridge fixing end |
| CN110084377A (en) * | 2019-04-30 | 2019-08-02 | 京东城市(南京)科技有限公司 | Method and apparatus for constructing decision tree |
| CN110175098A (en) * | 2019-02-03 | 2019-08-27 | 中国银联股份有限公司 | A kind of information processing method and information processing system |
| CN110221572A (en) * | 2019-04-26 | 2019-09-10 | 北京龙鼎源科技股份有限公司 | System control method and device |
| CN110245688A (en) * | 2019-05-21 | 2019-09-17 | 中国平安财产保险股份有限公司 | A kind of method and relevant apparatus of data processing |
| CN110245140A (en) * | 2019-06-12 | 2019-09-17 | 同盾控股有限公司 | Data branch mailbox processing method and processing device, electronic equipment and computer-readable medium |
| CN110384483A (en) * | 2019-07-23 | 2019-10-29 | 福建中医药大学 | Moving equilibrium appraisal procedure and terminal |
| CN110472659A (en) * | 2019-07-05 | 2019-11-19 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer readable storage medium and computer equipment |
| CN110598845A (en) * | 2019-08-13 | 2019-12-20 | 中国平安人寿保险股份有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN110688429A (en) * | 2019-08-14 | 2020-01-14 | 中国平安人寿保险股份有限公司 | Target employee screening method and device, computer equipment and storage medium |
| CN110706015A (en) * | 2019-08-21 | 2020-01-17 | 北京大学（天津滨海）新一代信息技术研究院 | Advertisement click rate prediction oriented feature selection method |
| CN110970100A (en) * | 2019-11-04 | 2020-04-07 | 广州金域医学检验中心有限公司 | Method, device and equipment for detecting item coding and computer readable storage medium |
| CN110990857A (en) * | 2019-12-11 | 2020-04-10 | 支付宝(杭州)信息技术有限公司 | Multi-party combined feature evaluation method and device for protecting privacy and safety |
| CN111221517A (en) * | 2019-10-12 | 2020-06-02 | 中国平安财产保险股份有限公司 | Model creating method and device, computer equipment and readable storage medium |
| CN111368901A (en) * | 2020-02-28 | 2020-07-03 | 深圳前海微众银行股份有限公司 | Multi-party combined modeling method, device and medium based on federal learning |
| CN111782900A (en) * | 2020-08-06 | 2020-10-16 | 平安银行股份有限公司 | Abnormal service detection method and device, electronic equipment and storage medium |
| CN111898765A (en) * | 2020-07-29 | 2020-11-06 | 深圳前海微众银行股份有限公司 | Feature binning method, device, equipment and readable storage medium |
| CN112183644A (en) * | 2020-09-29 | 2021-01-05 | 中国平安人寿保险股份有限公司 | Index stability monitoring method and device, computer equipment and medium |
| CN112367396A (en) * | 2020-11-06 | 2021-02-12 | 支付宝(杭州)信息技术有限公司 | Method and device for determining sample characteristic quantile in distributed cluster |
| CN112698988A (en) * | 2020-12-30 | 2021-04-23 | 安徽迪科数金科技有限公司 | Method for analyzing and processing super-large text file based on distributed system |
| CN113344626A (en) * | 2021-06-03 | 2021-09-03 | 上海冰鉴信息科技有限公司 | Data feature optimization method and device based on advertisement push |
| WO2021232588A1 (en) * | 2020-05-21 | 2021-11-25 | 平安国际智慧城市科技股份有限公司 | Food safety risk assessment method, apparatus, device, and storage medium |
| CN114329127A (en) * | 2021-12-30 | 2022-04-12 | 北京瑞莱智慧科技有限公司 | Characteristic box dividing method, device and storage medium |
| CN114496209A (en) * | 2022-02-18 | 2022-05-13 | 青岛市中心血站 | Blood donation intelligent decision method and system |
| CN114816506A (en) * | 2022-04-21 | 2022-07-29 | 北京金堤科技有限公司 | Model feature rapid processing method and device, storage medium and electronic equipment |
| CN115168848A (en) * | 2022-09-08 | 2022-10-11 | 南京鼎山信息科技有限公司 | Interception feedback processing method based on big data analysis interception |
| CN117408787A (en) * | 2023-12-15 | 2024-01-16 | 江西求是高等研究院 | Root cause mining analysis method and system based on decision tree |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070185896A1 (en) * | 2006-02-01 | 2007-08-09 | Oracle International Corporation | Binning predictors using per-predictor trees and MDL pruning |
| CN105786860A (en) * | 2014-12-23 | 2016-07-20 | 华为技术有限公司 | Data processing method and device in data modeling |
| CN106250461A (en) * | 2016-07-28 | 2016-12-21 | 北京北信源软件股份有限公司 | A kind of algorithm utilizing gradient lifting decision tree to carry out data mining based on Spark framework |
| CN107025228A (en) * | 2016-01-29 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method for recommending problem and equipment |
-
2018
- 2018-04-09 CN CN201810309823.0A patent/CN108764273B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070185896A1 (en) * | 2006-02-01 | 2007-08-09 | Oracle International Corporation | Binning predictors using per-predictor trees and MDL pruning |
| CN105786860A (en) * | 2014-12-23 | 2016-07-20 | 华为技术有限公司 | Data processing method and device in data modeling |
| CN107025228A (en) * | 2016-01-29 | 2017-08-08 | 阿里巴巴集团控股有限公司 | A kind of method for recommending problem and equipment |
| CN106250461A (en) * | 2016-07-28 | 2016-12-21 | 北京北信源软件股份有限公司 | A kind of algorithm utilizing gradient lifting decision tree to carry out data mining based on Spark framework |
Cited By (42)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109919783A (en) * | 2019-01-31 | 2019-06-21 | 德联易控科技(北京)有限公司 | Risk Identification Method, device, equipment and the storage medium of vehicle insurance Claims Resolution case |
| CN110175098A (en) * | 2019-02-03 | 2019-08-27 | 中国银联股份有限公司 | A kind of information processing method and information processing system |
| CN110175098B (en) * | 2019-02-03 | 2023-01-10 | 中国银联股份有限公司 | Information processing method and information processing system |
| CN109978056A (en) * | 2019-03-26 | 2019-07-05 | 广东工业大学 | A kind of Metro Passenger classification method based on machine learning |
| CN110070270A (en) * | 2019-03-29 | 2019-07-30 | 广东民航机场建设有限公司 | Transmission equipment control method and device for connecting bridge fixing end |
| CN110221572A (en) * | 2019-04-26 | 2019-09-10 | 北京龙鼎源科技股份有限公司 | System control method and device |
| CN110221572B (en) * | 2019-04-26 | 2021-04-20 | 北京龙鼎源科技股份有限公司 | System control method and device |
| CN110084377A (en) * | 2019-04-30 | 2019-08-02 | 京东城市(南京)科技有限公司 | Method and apparatus for constructing decision tree |
| CN110084377B (en) * | 2019-04-30 | 2023-09-29 | 京东城市(南京)科技有限公司 | Method and device for constructing decision tree |
| CN110245688A (en) * | 2019-05-21 | 2019-09-17 | 中国平安财产保险股份有限公司 | A kind of method and relevant apparatus of data processing |
| CN110245688B (en) * | 2019-05-21 | 2024-05-28 | 中国平安财产保险股份有限公司 | Data processing method and related device |
| CN110245140A (en) * | 2019-06-12 | 2019-09-17 | 同盾控股有限公司 | Data branch mailbox processing method and processing device, electronic equipment and computer-readable medium |
| WO2020248356A1 (en) * | 2019-06-12 | 2020-12-17 | 同盾控股有限公司 | Data binning processing method and apparatus, electronic device and computer-readable medium |
| CN110472659B (en) * | 2019-07-05 | 2024-03-08 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer readable storage medium and computer equipment |
| CN110472659A (en) * | 2019-07-05 | 2019-11-19 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer readable storage medium and computer equipment |
| CN110384483A (en) * | 2019-07-23 | 2019-10-29 | 福建中医药大学 | Moving equilibrium appraisal procedure and terminal |
| CN110598845B (en) * | 2019-08-13 | 2023-04-07 | 中国平安人寿保险股份有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN110598845A (en) * | 2019-08-13 | 2019-12-20 | 中国平安人寿保险股份有限公司 | Data processing method, data processing device, computer equipment and storage medium |
| CN110688429A (en) * | 2019-08-14 | 2020-01-14 | 中国平安人寿保险股份有限公司 | Target employee screening method and device, computer equipment and storage medium |
| CN110706015A (en) * | 2019-08-21 | 2020-01-17 | 北京大学（天津滨海）新一代信息技术研究院 | Advertisement click rate prediction oriented feature selection method |
| CN111221517A (en) * | 2019-10-12 | 2020-06-02 | 中国平安财产保险股份有限公司 | Model creating method and device, computer equipment and readable storage medium |
| CN110970100A (en) * | 2019-11-04 | 2020-04-07 | 广州金域医学检验中心有限公司 | Method, device and equipment for detecting item coding and computer readable storage medium |
| CN110990857A (en) * | 2019-12-11 | 2020-04-10 | 支付宝(杭州)信息技术有限公司 | Multi-party combined feature evaluation method and device for protecting privacy and safety |
| CN111368901A (en) * | 2020-02-28 | 2020-07-03 | 深圳前海微众银行股份有限公司 | Multi-party combined modeling method, device and medium based on federal learning |
| WO2021232588A1 (en) * | 2020-05-21 | 2021-11-25 | 平安国际智慧城市科技股份有限公司 | Food safety risk assessment method, apparatus, device, and storage medium |
| CN111898765A (en) * | 2020-07-29 | 2020-11-06 | 深圳前海微众银行股份有限公司 | Feature binning method, device, equipment and readable storage medium |
| CN111782900A (en) * | 2020-08-06 | 2020-10-16 | 平安银行股份有限公司 | Abnormal service detection method and device, electronic equipment and storage medium |
| CN111782900B (en) * | 2020-08-06 | 2024-03-19 | 平安银行股份有限公司 | Abnormal service detection method and device, electronic equipment and storage medium |
| CN112183644A (en) * | 2020-09-29 | 2021-01-05 | 中国平安人寿保险股份有限公司 | Index stability monitoring method and device, computer equipment and medium |
| CN112183644B (en) * | 2020-09-29 | 2024-05-03 | 中国平安人寿保险股份有限公司 | Index stability monitoring method and device, computer equipment and medium |
| CN112367396A (en) * | 2020-11-06 | 2021-02-12 | 支付宝(杭州)信息技术有限公司 | Method and device for determining sample characteristic quantile in distributed cluster |
| CN112698988A (en) * | 2020-12-30 | 2021-04-23 | 安徽迪科数金科技有限公司 | Method for analyzing and processing super-large text file based on distributed system |
| CN113344626A (en) * | 2021-06-03 | 2021-09-03 | 上海冰鉴信息科技有限公司 | Data feature optimization method and device based on advertisement push |
| CN114329127A (en) * | 2021-12-30 | 2022-04-12 | 北京瑞莱智慧科技有限公司 | Characteristic box dividing method, device and storage medium |
| CN114329127B (en) * | 2021-12-30 | 2023-06-20 | 北京瑞莱智慧科技有限公司 | Feature binning method, device and storage medium |
| CN114496209B (en) * | 2022-02-18 | 2022-09-27 | 青岛市中心血站 | Intelligent decision-making method and system for blood donation |
| CN114496209A (en) * | 2022-02-18 | 2022-05-13 | 青岛市中心血站 | Blood donation intelligent decision method and system |
| CN114816506A (en) * | 2022-04-21 | 2022-07-29 | 北京金堤科技有限公司 | Model feature rapid processing method and device, storage medium and electronic equipment |
| CN115168848B (en) * | 2022-09-08 | 2022-12-16 | 南京鼎山信息科技有限公司 | Interception feedback processing method based on big data analysis interception |
| CN115168848A (en) * | 2022-09-08 | 2022-10-11 | 南京鼎山信息科技有限公司 | Interception feedback processing method based on big data analysis interception |
| CN117408787A (en) * | 2023-12-15 | 2024-01-16 | 江西求是高等研究院 | Root cause mining analysis method and system based on decision tree |
| CN117408787B (en) * | 2023-12-15 | 2024-03-05 | 江西求是高等研究院 | Root cause mining analysis method and system based on decision tree |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108764273B (en) | 2023-12-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108764273A (en) | A kind of method, apparatus of data processing, terminal device and storage medium | |
| CN108733631A (en) | A kind of data assessment method, apparatus, terminal device and storage medium | |
| US9984336B2 (en) | Classification rule sets creation and application to decision making | |
| CN105893609A (en) | Mobile APP recommendation method based on weighted mixing | |
| CN110516910A (en) | Declaration form core based on big data protects model training method and core protects methods of risk assessment | |
| CN108446794A (en) | One kind being based on multiple convolutional neural networks combination framework deep learning prediction techniques | |
| CN107169628A (en) | A kind of distribution network reliability evaluation method based on big data mutual information attribute reduction | |
| Xue et al. | Optimizing ontology alignment through memetic algorithm based on partial reference alignment | |
| CN112036483B (en) | AutoML-based object prediction classification method, device, computer equipment and storage medium | |
| CN113822315A (en) | Attribute graph processing method and device, electronic equipment and readable storage medium | |
| CN116489038A (en) | Network traffic prediction method, device, equipment and medium | |
| CN109062867A (en) | Object and attribute while increased matrix Dynamic Attribute Reduction method | |
| CN114721898A (en) | Edge cloud server utilization rate prediction method and device based on boosting algorithm and storage medium | |
| Shi et al. | Human resources balanced allocation method based on deep learning algorithm | |
| CN111930944A (en) | File label classification method and device | |
| CN113656692B (en) | Product recommendation method, device, equipment and medium based on knowledge migration algorithm | |
| CN113779933A (en) | Commodity encoding method, electronic device and computer-readable storage medium | |
| CN109086373B (en) | Method for constructing fair link prediction evaluation system | |
| CN104615605B (en) | The method and apparatus of classification for prediction data object | |
| Luo et al. | E-commerce big data classification and mining algorithm based on artificial intelligence | |
| CN113190662A (en) | Topic segmentation method based on discourse structure diagram network | |
| CN111026661A (en) | Method and system for comprehensively testing usability of software | |
| Xie | Research on Weibo user behavior system for subjective perception and big data mining technology | |
| CN115238075B (en) | Text sentiment classification method based on hypergraph pooling | |
| CN112862536B (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |