CN109558952A - Data processing method, system, equipment and storage medium - Google Patents
Data processing method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN109558952A CN109558952A CN201811431899.7A CN201811431899A CN109558952A CN 109558952 A CN109558952 A CN 109558952A CN 201811431899 A CN201811431899 A CN 201811431899A CN 109558952 A CN109558952 A CN 109558952A
- Authority
- CN
- China
- Prior art keywords
- data
- training
- model
- cloud
- object module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
Data processing method, system, equipment and storage medium provided by the invention, belong to field of computer technology.The data processing method includes: acquisition training data, is iterated training to default training pattern according to the training data, obtains object module；The object module is disposed to cloud, to make to reduce the unobstructed complexity of maintenance whole process in the operation set of each step, and at the same time reducing exploitation, test, the communication cost between product personnel.
Description
Technical field
The present invention relates to field of computer technology, are situated between in particular to data processing method, system, equipment and storage
Matter.
Background technique
As AI algorithm, especially deep learning algorithm constantly penetrate into various industries, more and more AI products are continuous
It comes out.The characteristics of due to deep learning algorithm model: computational complexity is big and mass data is needed to participate in training, is mostly based on
The product of deep learning or service are all based on cloud.A cloud group of planes can guarantee the timeliness that model calculates, and to line last time
The timely cleaning treatment of the data of stream is sent into model repetitive exercise, and the ability and performance of deep learning model is continuously improved.However by
In the group of planes that is related to of repetitive exercise circulation is more, network environment is complicated, the dispersion of every single stepping, so that maintenance whole process is unobstructed
Complexity is higher, and exploitation is tested, links up increased costs between product personnel.
Summary of the invention
In view of this, provided in an embodiment of the present invention be designed to provide a kind of data processing method, system, equipment and deposit
Storage media.
To achieve the goals above, technical solution used in the embodiment of the present invention is as follows:
In a first aspect, a kind of data processing method provided in an embodiment of the present invention, comprising: obtain training data；According to institute
It states training data and training is iterated to default training pattern, obtain object module；The object module is sent to cloud, with
Run the object module in the cloud.
With reference to first aspect, described to obtain the embodiment of the invention provides the first possible embodiment of first aspect
Take training data, comprising: obtain the data acquisition request of user's input；The data acquisition request is sent to cloud；It receives
The cloud return with the matched request data of the data acquisition request institute；The request data is labeled, is obtained
The training data.
The possible embodiment of with reference to first aspect the first, the embodiment of the invention provides second of first aspect
Possible embodiment, it is described that the data acquisition request is sent to cloud, comprising: to be based on prefixed time interval for the number
It is sent to cloud according to acquisition request, so that the cloud returns and the matched request data of the data acquisition request institute.
The possible embodiment of with reference to first aspect the first, the embodiment of the invention provides the third of first aspect
Possible embodiment, it is described that the request data is labeled, obtain the training data, comprising: obtain user's input
Filtration parameter；The request data is filtered according to the filtration parameter, obtains filtered request data to be processed；
The request data to be processed is labeled, the training data is obtained.
The third possible embodiment with reference to first aspect, the embodiment of the invention provides the 4th kind of first aspect
Possible embodiment, it is described that the request data to be processed is labeled, obtain the training data, comprising: obtain institute
State the corresponding markup information of request data to be processed；The request data to be processed is labeled according to the markup information,
Obtain labeled data；It obtains and update information corresponding to the labeled data；According to the update information to the mark number
According to being modified, revised target labeled data is obtained；Using the target labeled data as the training data.
With reference to first aspect, the embodiment of the invention provides the 5th kind of possible embodiment of first aspect, described
Training is iterated to default training pattern according to the training data, obtains object module, comprising: according to the training data pair
Default training pattern is iterated training；Determine whether to reach the preset condition for exporting trained model；If so, by the institute
Trained model is as object module.
The 5th kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 6th kind of first aspect
Possible embodiment, it is described to determine whether to reach the preset condition for exporting trained model, comprising: to determine that "current" model is instructed
Whether experienced training duration matches with preset model output time interval；If so, characterization has reached the preset condition.
The 5th kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 7th kind of first aspect
Possible embodiment, it is described to determine whether to reach the preset condition for exporting trained model, comprising: to determine that "current" model is instructed
Whether experienced the number of iterations is more than preset threshold；If so, characterization has reached the preset condition.
The 5th kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 8th kind of first aspect
Possible embodiment, further includes: if the preset condition of not up to the trained model of output, executes again according to the instruction
Practice data and is iterated trained step to default training pattern.
With reference to first aspect, the embodiment of the invention provides the 9th kind of possible embodiment of first aspect, described
Training is iterated to default training pattern according to the training data, obtains object module, comprising: according to the training data pair
Default training pattern is iterated training, obtains multiple training patterns；Obtain accuracy rate corresponding to each training pattern；
The optimal model of a training is determined from how much training patterns according to the accuracy rate；The optimal model of the training is made
For object module.
With reference to first aspect, the embodiment of the invention provides the tenth kind of possible embodiments of first aspect, described to incite somebody to action
The object module is sent to cloud, comprising: obtains the corresponding survey of multiple object modules institutes in preset time period
Test result；It is determined according to multiple test results to deployment model；Cloud is sent to deployment model by described.
The tenth kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 10th of first aspect the
A kind of possible embodiment, it is described to be determined according to multiple test results to deployment model, comprising: by multiple surveys
Test result is compared, and obtains comparison information；Determine that one is optimal from multiple test results according to the comparison information
Target detection result；Using the object module corresponding to the target detection result as to deployment model.
The tenth kind of possible embodiment with reference to first aspect, the embodiment of the invention provides the 10th of first aspect the
Two kinds of possible embodiments, it is described to be determined according to multiple test results to deployment model, comprising: to obtain user's input
Assessment instruction；Each test result is assessed according to assessment instruction, obtains assessment score；According to institute's commentary
Estimate score to determine from multiple object modules one to deployment model.
With reference to first aspect, the embodiment of the invention provides the 13rd kind of possible embodiment of first aspect, institutes
State method further include: obtain the model data request instruction of user's input；The model data request instruction is sent to described
Cloud；Receive that the cloud returns with the model data request instruction the matched object module version information with
And the performance information of the object module.
Second aspect, a kind of data processing system provided in an embodiment of the present invention, the data processing system includes data
Acquiring unit, for obtaining training data；Model training unit, for being carried out according to the training data to default training pattern
Repetitive exercise obtains object module；Deployment unit, for the object module to be sent to cloud, so that the object module
It is run in the cloud.
The third aspect, a kind of terminal device provided in an embodiment of the present invention, comprising: memory, processor and be stored in
In the memory and the computer program that can run on the processor, when the processor executes the computer program
It realizes as described in any one of first aspect the step of data processing method.
Fourth aspect, a kind of storage medium provided in an embodiment of the present invention are stored with instruction on the storage medium, work as institute
Instruction is stated when running on computers, so that the computer executes such as the described in any item data processing methods of first aspect.
Compared with prior art, the embodiment of the present invention bring it is following the utility model has the advantages that
Data processing method, system, equipment and storage medium provided in an embodiment of the present invention, by obtaining training data,
Training is iterated to default training pattern according to the training data, obtains object module；The object module is sent to
Cloud, to make in the operation set of each step, it is logical to reduce maintenance whole process so that the object module is run in the cloud
Smooth complexity, and at the same time reducing exploitation, test, the communication cost between product personnel.
Other feature and advantage of the disclosure will illustrate in the following description, alternatively, Partial Feature and advantage can be with
Deduce from specification or unambiguously determine, or by implement the disclosure above-mentioned technology it can be learnt that.
To enable the above objects, features and advantages of the present invention to be clearer and more comprehensible, preferred embodiment is cited below particularly, and cooperate
Appended attached drawing, is described in detail below.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 is the flow chart for the data processing method that first embodiment of the invention provides；
Fig. 2 is that interface schematic diagram is disposed in the model measurement in data processing method shown in FIG. 1；
Fig. 3 is the assessment interface schematic diagram in data processing method shown in FIG. 1；
Fig. 4 is the performance information display interface schematic diagram in data processing method shown in FIG. 1；
Fig. 5 is the version information display interface schematic diagram in data processing method shown in FIG. 1；
Fig. 6 is the functional block diagram for the data processing system that second embodiment of the invention provides；
Fig. 7 is a kind of schematic diagram for terminal device that third embodiment of the invention provides.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.Therefore,
The model of claimed invention is not intended to limit to the detailed description of the embodiment of the present invention provided in the accompanying drawings below
It encloses, but is merely representative of selected embodiment of the invention.Based on the embodiments of the present invention, those of ordinary skill in the art are not having
Every other embodiment obtained under the premise of creative work is made, shall fall within the protection scope of the present invention.
With reference to the accompanying drawing, it elaborates to some embodiments of the present invention.In the absence of conflict, following
Feature in embodiment and embodiment can be combined with each other.
First embodiment
The group of planes being related to due to existing repetitive exercise circulation is more, network environment is complicated, the dispersion of every single stepping, so that
Safeguard that the unobstructed complexity of whole process is higher, exploitation is tested, links up increased costs between product personnel, complete in order to reduce maintenance
The unobstructed complexity of process and reduction exploitation are tested, the communication cost between product personnel, and the present embodiment provides firstly one
Kind data processing method, it should be noted that step shown in the flowchart of the accompanying drawings can be held in such as one group of computer
It is executed in the computer system of row instruction, although also, logical order is shown in flow charts, in some cases,
It can be with the steps shown or described are performed in an order that is different from the one herein.It describes in detail below to the present embodiment.
Referring to Fig. 1, being the flow chart of data processing method provided in an embodiment of the present invention.It below will be to shown in FIG. 1
Detailed process is described in detail.
Step S101 obtains training data.
As an implementation, step S101 includes: the data acquisition request for obtaining user's input；The data are obtained
Request is taken to be sent to cloud；Receive that the cloud returns with the matched request data of the data acquisition request institute；To described
Request data is labeled, and obtains the training data.
Optionally, the data acquisition request can be based on ftp (File Transfer Protocol, file transmission
Agreement), sftp (Secure File Transfer Protocol, secure file transportation protocol), http (Hyper Text
Transport Protocol, hypertext transfer protocol) etc. agreements the data acquisition request is sent to cloud, thus from cloud
Downloading and the matched request data of the data acquisition request institute.
Optionally, request data includes but is not limited to image, video, text, audio etc. and required parameter information.
Optionally, request data is done into persistence processing, deposit includes but is not limited to relevant database, non-relational number
Stored according to library or object etc..
Optionally, described that the data acquisition request is sent to cloud, comprising: to be based on prefixed time interval for the number
Cloud is sent to according to acquisition request.
Optionally, the setting of prefixed time interval can be configured according to user demand.Here, being not especially limited.
For example, grabbing a request data from cloud every 3 days.
Optionally, described that the request data is labeled, obtain the training data, comprising: obtain user's input
Filtration parameter；The request data is filtered according to the filtration parameter, obtains filtered request data to be processed；
The request data to be processed is labeled, the training data is obtained.
Optionally, the filtration parameter can be the time, be also possible to quantity etc..It is asked for example, user needs to obtain 50
Data or user are asked to need to obtain XX days request datas.
Optionally, the request data is labeled, obtains the training data, comprising: according to preset strategy from asking
It asks data screening to go out the request data in objective time interval, obtains request data to be processed；The request data to be processed is carried out
Mark, obtains the training data.
Optionally, preset strategy can be but not limited to random sampling, based on grading parameters ranked-set on line etc..
Optionally, objective time interval is the period of user's input.For example, user needs to filter out XX days request datas.
Optionally, described that the request data to be processed is labeled, obtain the training data, comprising: obtain institute
State the corresponding markup information of request data to be processed；The request data to be processed is labeled according to the markup information,
Obtain labeled data；It obtains and update information corresponding to the labeled data；According to the update information to the mark number
According to being modified, revised target labeled data is obtained；Using the target labeled data as the training data.
Optionally, markup information can be user's input, be also possible to machine generation.Here, being not especially limited.
Optionally, update information can be what user was inputted based on labeled data.
By being modified to labeled data, it can provide quality higher training data for subsequent model training, into
And make the model trained more excellent.
Step S102 is iterated training to default training pattern according to the training data, obtains target mould.
As an implementation, step S102 includes: to be iterated according to the training data to default training pattern
Training；Determine whether to reach the preset condition for exporting trained model；If so, using the trained model as target mould
Type.
Optionally, described to determine whether to reach the preset condition for exporting trained model, comprising: to determine that "current" model is instructed
Whether experienced training duration matches with preset model output time interval；If so, characterization has reached the preset condition.
Wherein, the training duration of "current" model training refers to the duration from last output model and current time.For example,
The last model output time is t1, current time t2, then a length of t2-t1 when the training of "current" model training.
Optionally, the setting of preset model output time interval can be configured according to user's actual need, for example, can
To be either 24 hours etc. 12 hours.Here, being not especially limited.
Wherein, matching refers to that the training duration of "current" model training is equal to preset model output time interval.
Optionally, described to determine whether to reach the preset condition for exporting trained model, comprising: to determine that "current" model is instructed
Whether experienced the number of iterations is more than preset threshold；If so, characterization has reached the preset condition.
Optionally, the setting of preset threshold can be configured according to user's actual need, for example, it may be 12 times or
It is 15 inferior.Here, being not especially limited.
In the present embodiment, by determining whether to reach the preset condition for exporting trained model, thus to judge to be
The trained model of no output, thus with the system decoupling using the model is needed, and then improve robustness.
In a possible embodiment, the method also includes: if the preset condition of not up to the trained model of output,
It then executes again and training is iterated to default training pattern according to the training data.
As another embodiment, step S102 includes: to be changed according to the training data to default training pattern
Generation training, obtains multiple training patterns；Obtain accuracy rate corresponding to each training pattern；It is multiple according to the accuracy rate
The optimal model of a training is determined in the training pattern；Using the optimal model of the training as object module.
Optionally, it in repetitive exercise, can be spaced to obtain a model at regular intervals, and the time interval is less than
Preset model output time interval.
Optionally, the setting of the certain time interval can be configured according to trained demand.Here, not limiting specifically
It is fixed.
Certainly, in actual use, the excellent of model training can also be determined by the precision and/or recall rate of model
Effect；Either determined by positive sample score and negative sample score.In general, the setting of precision and/or recall rate can be with
It is configured according to user demand, for example, precision and/or recall rate can be 98%.Generally, precision and/or recall rate
It is more high, indicate the better of model training.
The object module is sent to cloud by step S103.
As an implementation, step S103 includes: that the multiple object modules obtained in preset time period divide
Not corresponding test result；It is determined according to multiple test results to deployment model；It is sent to described to deployment model
Cloud.
Wherein, the setting of preset time period can be configured according to user's actual need, for example, it may be 12 hours or
Person is 24 hours etc..Here, being not especially limited.For example, being tested together all object modules obtained in 12 hours.
Optionally, test result includes but is not limited to object module tested date, positive sample score, negative sample score
Deng.
For example, as shown in Fig. 2, in model measurement deployment interface, when the user clicks when model measurement deployment button,
Interface is disposed in model measurement can be according to test result displaying target model tested date, positive sample score, negative sample score
Deng tester being checked to the test result generated after the test of certain day object module in the interface.
Optionally, described to be determined according to multiple test results to deployment model, comprising: to tie multiple tests
Fruit is compared, and obtains comparison information；An optimal target is determined from multiple test results according to the comparison information
Test result；Using the object module corresponding to the target detection result as to deployment model.
Optionally, the conclusion obtained afterwards is compared to each object module for characterizing in comparison information.For example,
Assuming that there is tri- object modules of A, B, C, corresponding test result is respectively a, b, c, after a, b, c are compared, finds A's
Positive sample score is greater than the positive sample score of two object modules of B, C, and the negative sample score of A is lower than two object modules of B, C
Negative sample score, then using a as target detection as a result, using the corresponding object module A of a as to deployment model.
Optionally, described to be determined according to multiple test results to deployment model, comprising: to obtain commenting for user's input
Estimate instruction；Each test result is assessed according to assessment instruction, obtains assessment score；According to the assessment point
Several determinations one from multiple object modules are to deployment model.
Continue for by taking above-mentioned example as an example, if it is further right also to think on the basis of having checked information as shown in Figure 2
Certain data are verified, then user can generate assessment instruction, to make to each described by click " model evaluation " button
Test result is assessed, and disposes the interface display assessment score by model measurement.As shown in figure 3, model measurement is disposed
Interface can show each object module and with assessment score corresponding to each object module.For example, in Fig. 3
Show assessment score corresponding to model 1, assessment score corresponding to model 2 etc..
Optionally, cloud is sent to deployment model by described, comprising: obtain the hair version instruction of user's input；According to institute
It states hair version instruction and is sent to cloud to deployment model for described.
Optionally, the instruction of hair version includes the corresponding version information to deployment model, and hair version instruction is for corresponding by its
Version to deployment model is issued.
As shown in Fig. 2, corresponding to " hair version " button in column by point percussion version key to generate hair version instruction, Jin Ergen
Issuing to deployment model to the version is instructed according to hair version.
In actual use, general by monitoring whether " hair version " button is clicked, thus when being monitored to click, it is raw
At hair version instruction.For example, user click or double-click " hair version " button are to generate hair version instruction.
In a possible embodiment, data processing method further include: obtain the model data request instruction of user's input；
The model data request instruction is sent to the cloud；It is returning with the model data request instruction to receive the cloud
The matched object module performance information.
Optionally, performance information includes but is not limited to general offensive number, missing inspection, omission factor, erroneous detection and service request sum
Deng.
Optionally, model data request instruction includes the instruction of online service model monitoring and model version trace instruction.
Optionally, the online service model monitoring instruction of user's input is obtained；The online service model monitoring is instructed
It is sent to cloud；Receive the performance information with the object module of online service model monitoring instruction that cloud returns；It obtains
The model version trace instruction of user's input；The model version trace instruction is sent to cloud；Receive cloud return with
The version information of the object module of the model version trace instruction.
For example, as shown in figure 4, when the user clicks " online service model monitoring " button when, generate online service mould
It is deployed to grab current time according to the instruction of online service model monitoring in real time from the online service for type Monitoring instruction
The performance information of object module, and the performance information of the model is back to the operation interface and is shown.For example, Fig. 4
In shown date be xx xx month x1 day, mark progress for 61% object module performance information are as follows: general offensive number is
514, missing inspection 5, omission factor 0.097%, erroneous detection 6 and service request sum are 245939.Optionally, when user's point
When hitting " request details " button, automatic paging face shows details.
In another example operation interface as shown in Figure 5, when the user clicks when " model version tracking " button, generates model version
This trace instruction grabs current time deployed object module according to model version trace instruction in real time from the cloud
Model version, and the model version is returned and is shown.For example, group of planes information shown in Fig. 5 is a group of planes _ 1, machine
Group is abbreviated as MG_1, and the object module that corresponding model version is V5.0.1.Optionally, " group of planes mould is checked when the user clicks
Type requests details key " when, automatic paging face shows details.
Data processing method provided by the embodiment of the present invention, by obtaining training data, according to the training data pair
Default training pattern is iterated training, obtains object module；The object module is sent to cloud, so that the target mould
Type is run in the cloud, thus make to reduce the unobstructed complexity of maintenance whole process in the operation set of each step, and at the same time
Reduce exploitation, test, the communication cost between product personnel.
Second embodiment
Corresponding to the data processing method in first embodiment, Fig. 6 is shown using at data shown in first embodiment
The one-to-one data processing system of reason method.As shown in fig. 6, the data processing system 400 includes data capture unit
410, model training unit 420 and deployment unit 430.Wherein, data capture unit 410, model training unit 420 and deployment are single
The realization function of member 430 is gathered with step corresponding in first embodiment to be corresponded, and to avoid repeating, the present embodiment is not detailed one by one
It states.
Data capture unit 410, for obtaining training data.
Optionally, data capture unit 410 are also used to obtain the data acquisition request of user's input；The data are obtained
Request is taken to be sent to cloud；Receive that the cloud returns with the matched request data of the data acquisition request institute.
Optionally, the data acquisition request is sent to cloud, comprising: the data are obtained based on prefixed time interval
Request is taken to be sent to cloud.
Optionally, described that the request data is labeled, obtain the training data, comprising: obtain user's input
Filtration parameter；The request data is filtered according to the filtration parameter, obtains filtered request data to be processed；
The request data to be processed is labeled, the training data is obtained.
Optionally, described that the request data to be processed is labeled, obtain the training data, comprising: obtain institute
State the corresponding markup information of request data to be processed；The request data to be processed is labeled according to the markup information,
Obtain labeled data；It obtains and update information corresponding to the labeled data；According to the update information to the mark number
According to being modified, revised target labeled data is obtained；Using the target labeled data as the training data.
Model training unit 420 obtains mesh for being iterated training to default training pattern according to the training data
Mark model.
Optionally, model training unit 420 is also used to be iterated instruction to default training pattern according to the training data
Practice；Determine whether to reach the preset condition for exporting trained model；If so, using the trained model as target mould
Type.
Optionally, described to determine whether to reach the preset condition for exporting trained model, comprising: to determine that "current" model is instructed
Whether experienced training duration matches with preset model output time interval；If so, characterization has reached the preset condition.
Optionally, described to determine whether to reach the preset condition for exporting trained model, comprising: to determine that "current" model is instructed
Whether experienced the number of iterations is more than preset threshold；If so, characterization has reached the preset condition.
In a possible embodiment, data processing system 400 further include: execution unit, execution unit, if for not reaching
To the preset condition for exporting trained model, then executes and default training pattern is iterated again according to the training data
Trained step.
Optionally, model training unit 420, is also used to: being iterated according to the training data to default training pattern
Training, obtains multiple training patterns；Obtain accuracy rate corresponding to each training pattern；According to the accuracy rate from multiple
The optimal model of a training is determined in the training pattern；Using the optimal model of the training as object module.
Deployment unit 430, for the object module to be sent to cloud, so that the object module is transported in the cloud
Row.
Optionally, deployment unit 430 are also used to obtain multiple object modules in preset time period and are respectively corresponded
Test result；It is determined according to multiple test results to deployment model；Cloud is sent to deployment model by described.
Optionally, described to be determined according to multiple test results to deployment model, comprising: to tie multiple tests
Fruit is compared, and obtains comparison information；An optimal target is determined from multiple test results according to the comparison information
Test result；Using the object module corresponding to the target detection result as to deployment model.
Optionally, described to be determined according to multiple test results to deployment model, comprising: to obtain commenting for user's input
Estimate instruction；Each test result is assessed according to assessment instruction, obtains assessment score；According to the assessment point
Several determinations one from multiple object modules are to deployment model.
In a possible embodiment, data processing system 400 further include: monitoring unit, monitoring unit are used for obtaining
The model data request instruction of family input；The model data request instruction is sent to the cloud；The cloud is received to return
Return with the model data request instruction the version information of the matched object module and the property of the object module
It can information.
3rd embodiment
As shown in fig. 7, being the schematic diagram of terminal device 600.The terminal device 600 includes memory 602, processor
604 and it is stored in the computer program 603 that can be run in the memory 602 and on the processor 604, the calculating
The data processing method in first embodiment is realized when machine program 603 is executed by processor 604, to avoid repeating, herein
It repeats no more.Alternatively, being realized when the computer program 603 is executed by processor 604 at the data in second embodiment
The function of each module/unit in reason system, to avoid repeating, details are not described herein again.
Illustratively, computer program 603 can be divided into one or more module/units, one or more mould
Block/unit is stored in memory 602, and is executed by processor 604, to complete the present invention.One or more module/units
It can be the series of computation machine program instruction section that can complete specific function, the instruction segment is for describing computer program 603
Implementation procedure in terminal device 600.For example, the data that computer program 603 can be divided into second embodiment obtain
Unit 410, model training unit 420 and deployment unit 430 are taken, the concrete function of each module is as described in second embodiment, herein
It does not repeat one by one.
Terminal device 600 can be desktop PC, notebook, palm PC and management server etc. and calculate equipment.
Wherein, memory 602 may be, but not limited to, random access memory (Random Access Memory,
RAM), read-only memory (Read Only Memory, ROM), programmable read only memory (Programmable Read-
Only Memory, PROM), erasable read-only memory (Erasable Programmable Read-Only Memory,
EPROM), electricallyerasable ROM (EEROM) (Electric Erasable Programmable Read-Only Memory,
EEPROM) etc..Wherein, memory 602 is for storing program, and the processor 604 is after receiving and executing instruction, described in execution
The method of program, the flow definition that aforementioned any embodiment of the embodiment of the present invention discloses can be applied in processor 604, or
It is realized by processor 604.
Processor 604 may be a kind of IC chip, the processing capacity with signal.Above-mentioned processor 604 can
To be general processor, including central processing unit (Central Processing Unit, CPU), network processing unit (Network
Processor, NP) etc.；It can also be digital signal processor (Digital Signal Processor, DSP), dedicated integrated
Circuit (Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components.It may be implemented or execute disclosed each method, step and the logic diagram in the embodiment of the present invention.It is general
Processor can be microprocessor or the processor is also possible to any conventional processor etc..
It is understood that structure shown in Fig. 7 is only a kind of structural schematic diagram of terminal device 600, terminal device 600
It can also include than more or fewer components shown in Fig. 7.Each component shown in fig. 7 can use hardware, software or its group
It closes and realizes.
Fourth embodiment
The embodiment of the present invention also provides a kind of storage medium, and instruction is stored on the storage medium, when described instruction exists
The data processing side in first embodiment is realized when running on computer, when the computer program is executed by processor
Method, to avoid repeating, details are not described herein again.Alternatively, realizing second embodiment institute when the computer program is executed by processor
The function of each module/unit in data processing system is stated, to avoid repeating, details are not described herein again.
Through the above description of the embodiments, those skilled in the art can be understood that the present invention can lead to
Hardware realization is crossed, the mode of necessary general hardware platform can also be added to realize by software, based on this understanding, this hair
Bright technical solution can be embodied in the form of software products, which can store in a non-volatile memories
In medium (can be CD-ROM, USB flash disk, mobile hard disk etc.), including some instructions are used so that computer equipment (can be with
It is personal computer, server or the network equipment etc.) method that executes each implement scene of the present invention.
The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention, for the skill of this field
For art personnel, the invention may be variously modified and varied.All within the spirits and principles of the present invention, made any to repair
Change, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.It should also be noted that similar label and letter exist
Similar terms are indicated in following attached drawing, therefore, once being defined in a certain Xiang Yi attached drawing, are then not required in subsequent attached drawing
It is further defined and explained.
Claims (17)
1. a kind of data processing method characterized by comprising
Obtain training data；
Training is iterated to default training pattern according to the training data, obtains object module；
The object module is sent to cloud, so that the object module is run in the cloud.
2. the method according to claim 1, wherein the acquisition training data, comprising:
Obtain the data acquisition request of user's input；
The data acquisition request is sent to cloud；
Receive that the cloud returns with the matched request data of the data acquisition request institute；
The request data is labeled, the training data is obtained.
3. according to the method described in claim 2, it is characterized in that, described be sent to cloud for the data acquisition request, packet
It includes:
The data acquisition request is sent to cloud based on prefixed time interval.
4. according to the method described in claim 2, obtaining described it is characterized in that, described be labeled the request data
Training data, comprising:
Obtain the filtration parameter of user's input；
The request data is filtered according to the filtration parameter, obtains filtered request data to be processed；
The request data to be processed is labeled, the training data is obtained.
5. according to the method described in claim 4, obtaining it is characterized in that, described be labeled the request data to be processed
To the training data, comprising:
Obtain the corresponding markup information of the request data to be processed；
The request data to be processed is labeled according to the markup information, obtains labeled data；
It obtains and update information corresponding to the labeled data；
The labeled data is modified according to the update information, obtains revised target labeled data；
Using the target labeled data as the training data.
6. the method according to claim 1, wherein it is described according to the training data to default training pattern into
Row iteration training, obtains object module, comprising:
Training is iterated to default training pattern according to the training data；
Determine whether to reach the preset condition for exporting trained model；
If so, using the trained model as object module.
7. according to the method described in claim 6, it is characterized in that, described determine whether to reach the pre- of the trained model of output
If condition, comprising:
Determine whether the training duration of "current" model training matches with preset model output time interval；
If so, characterization has reached the preset condition.
8. according to the method described in claim 6, it is characterized in that, described determine whether to reach the pre- of the trained model of output
If condition, comprising:
Whether the number of iterations for determining "current" model training is more than preset threshold；
If so, characterization has reached the preset condition.
9. according to the method described in claim 6, it is characterized by further comprising:
If the preset condition of not up to the trained model of output, executes again according to the training data to default trained mould
Type is iterated trained step.
10. the method according to claim 1, wherein it is described according to the training data to default training pattern
It is iterated training, obtains object module, comprising:
Training is iterated to default training pattern according to the training data, obtains multiple training patterns；
Obtain accuracy rate corresponding to each training pattern；
The optimal model of a training is determined from multiple training patterns according to the accuracy rate；
Using the optimal model of the training as object module.
11. the method according to claim 1, wherein described be sent to cloud for the object module, comprising:
Obtain the corresponding test result of multiple object modules institutes in preset time period；
It is determined according to multiple test results to deployment model；
Cloud is sent to deployment model by described.
12. according to the method for claim 11, which is characterized in that described to be determined according to multiple test results to portion
Affix one's name to model, comprising:
Multiple test results are compared, comparison information is obtained；
An optimal target detection result is determined from multiple test results according to the comparison information；
Using the object module corresponding to the target detection result as to deployment model.
13. according to the method for claim 11, which is characterized in that described to be determined according to multiple test results to portion
Affix one's name to model, comprising:
Obtain the assessment instruction of user's input；
Each test result is assessed according to assessment instruction, obtains assessment score；
It is determined from multiple object modules one to deployment model according to the assessment score.
14. the method according to claim 1, wherein the method also includes:
Obtain the model data request instruction of user's input；The model data request instruction is sent to the cloud；
Receive that the cloud returns with the model data request instruction the matched object module version information with
And the performance information of the object module.
15. a kind of data processing system characterized by comprising
Data capture unit, for obtaining training data；
Model training unit obtains object module for being iterated training to default training pattern according to the training data；
Deployment unit, for the object module to be sent to cloud, so that the object module is run in the cloud.
16. a kind of terminal device characterized by comprising memory, processor and storage are in the memory and can be
The computer program run on the processor, the processor realized when executing the computer program as claim 1 to
The step of 14 described in any item data processing methods.
17. a kind of storage medium, which is characterized in that instruction is stored on the storage medium, when described instruction on computers
When operation, so that the computer executes such as the described in any item data processing methods of claim 1 to 14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811431899.7A CN109558952A (en) | 2018-11-27 | 2018-11-27 | Data processing method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811431899.7A CN109558952A (en) | 2018-11-27 | 2018-11-27 | Data processing method, system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109558952A true CN109558952A (en) | 2019-04-02 |
Family
ID=65867811
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811431899.7A Pending CN109558952A (en) | 2018-11-27 | 2018-11-27 | Data processing method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109558952A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110245985A (en) * | 2019-06-11 | 2019-09-17 | 深圳前海微众银行股份有限公司 | A kind of information processing method and device |
| CN110286587A (en) * | 2019-05-21 | 2019-09-27 | 平安银行股份有限公司 | Method, server and the computer readable storage medium of implementation model iteration control |
| CN111061852A (en) * | 2019-12-13 | 2020-04-24 | 上海中通吉网络技术有限公司 | Data processing method, device and system |
| CN111178517A (en) * | 2020-01-20 | 2020-05-19 | 上海依图网络科技有限公司 | Model deployment method, system, chip, electronic device and medium |
| CN111859592A (en) * | 2019-04-08 | 2020-10-30 | 阿里巴巴集团控股有限公司 | Parameter determination method and device, electronic equipment and computer-readable storage medium |
| CN112257733A (en) * | 2019-10-25 | 2021-01-22 | 北京沃东天骏信息技术有限公司 | Model iteration method, second electronic device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106951925A (en) * | 2017-03-27 | 2017-07-14 | 成都小多科技有限公司 | Data processing method, device, server and system |
| CN108009589A (en) * | 2017-12-12 | 2018-05-08 | 腾讯科技（深圳）有限公司 | Sample data processing method, device and computer-readable recording medium |
| CN108108821A (en) * | 2017-12-29 | 2018-06-01 | 广东欧珀移动通信有限公司 | Model training method and device |
| CN108427939A (en) * | 2018-03-30 | 2018-08-21 | 百度在线网络技术（北京）有限公司 | model generating method and device |
| CN108718249A (en) * | 2018-04-27 | 2018-10-30 | 广州西麦科技股份有限公司 | Network accelerating method, device based on SDN network and computer readable storage medium |
-
2018
- 2018-11-27 CN CN201811431899.7A patent/CN109558952A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106951925A (en) * | 2017-03-27 | 2017-07-14 | 成都小多科技有限公司 | Data processing method, device, server and system |
| CN108009589A (en) * | 2017-12-12 | 2018-05-08 | 腾讯科技（深圳）有限公司 | Sample data processing method, device and computer-readable recording medium |
| CN108108821A (en) * | 2017-12-29 | 2018-06-01 | 广东欧珀移动通信有限公司 | Model training method and device |
| CN108427939A (en) * | 2018-03-30 | 2018-08-21 | 百度在线网络技术（北京）有限公司 | model generating method and device |
| CN108718249A (en) * | 2018-04-27 | 2018-10-30 | 广州西麦科技股份有限公司 | Network accelerating method, device based on SDN network and computer readable storage medium |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111859592A (en) * | 2019-04-08 | 2020-10-30 | 阿里巴巴集团控股有限公司 | Parameter determination method and device, electronic equipment and computer-readable storage medium |
| CN110286587A (en) * | 2019-05-21 | 2019-09-27 | 平安银行股份有限公司 | Method, server and the computer readable storage medium of implementation model iteration control |
| CN110286587B (en) * | 2019-05-21 | 2022-02-22 | 平安银行股份有限公司 | Method, server and computer readable storage medium for realizing model iteration control |
| CN110245985A (en) * | 2019-06-11 | 2019-09-17 | 深圳前海微众银行股份有限公司 | A kind of information processing method and device |
| CN110245985B (en) * | 2019-06-11 | 2023-09-08 | 深圳前海微众银行股份有限公司 | Information processing method and device |
| CN112257733A (en) * | 2019-10-25 | 2021-01-22 | 北京沃东天骏信息技术有限公司 | Model iteration method, second electronic device and storage medium |
| CN112257733B (en) * | 2019-10-25 | 2024-04-09 | 北京沃东天骏信息技术有限公司 | Model iteration method, second electronic equipment and storage medium |
| CN111061852A (en) * | 2019-12-13 | 2020-04-24 | 上海中通吉网络技术有限公司 | Data processing method, device and system |
| CN111178517A (en) * | 2020-01-20 | 2020-05-19 | 上海依图网络科技有限公司 | Model deployment method, system, chip, electronic device and medium |
| CN111178517B (en) * | 2020-01-20 | 2023-12-05 | 上海依图网络科技有限公司 | Model deployment method, system, chip, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109558952A (en) | Data processing method, system, equipment and storage medium | |
| JP6636071B2 (en) | Computer mounting method, computer system and computer device | |
| Rebert et al. | Optimizing seed selection for fuzzing | |
| CN104899663B (en) | A kind of data predication method and device | |
| Vanhoucke et al. | An overview of project data for integrated project management and control | |
| US9047423B2 (en) | Monte-Carlo planning using contextual information | |
| CN105045831B (en) | A kind of information push method and device | |
| CN104350471B (en) | Method and system for detecting anomalies in real-time in processing environment | |
| CN109587008A (en) | Detect the method, apparatus and storage medium of abnormal flow data | |
| CN107992410B (en) | Software quality monitoring method and device, computer equipment and storage medium | |
| CN113010389A (en) | Training method, fault prediction method, related device and equipment | |
| CN113660225A (en) | Network attack event prediction method, system, device and medium based on time sequence point | |
| Ridge | Design of experiments for the tuning of optimisation algorithms | |
| US7647130B2 (en) | Real-time predictive time-to-completion for variable configure-to-order manufacturing | |
| CN115237804A (en) | Performance bottleneck assessment method, performance bottleneck assessment device, electronic equipment, medium and program product | |
| Dhavachelvan et al. | Multi-agent-based integrated framework for intra-class testing of object-oriented software | |
| CN110059083A (en) | A kind of data evaluation method, apparatus and electronic equipment | |
| Robinson | Automated analysis of simulation output data | |
| CN103605670B (en) | A kind of method and apparatus for determining the crawl frequency of network resource point | |
| EP4169223A1 (en) | Method and apparatus to detect scripted network traffic | |
| US8543524B1 (en) | Method for benchmarking combinatorial problems and their solutions | |
| Hoos et al. | Empirical Analysis of Randomised Algorithms | |
| Li et al. | Search-based uncertainty-wise requirements prioritization | |
| CN113986704A (en) | TS-Decomposition-based data center high-frequency fault time domain early warning method and system | |
| Tian et al. | Analyzing and improving reliability: A tree-based approach |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |