CN110163248A

CN110163248A - Method for visualizing, device, computer equipment and the storage medium of model evaluation

Info

Publication number: CN110163248A
Application number: CN201910278714.1A
Authority: CN
Inventors: 陈飞; 彭绍东; 黎伟杰; 韩旭
Original assignee: Wenyuan Zhixing Co Ltd
Current assignee: Wenyuan Zhixing Co Ltd
Priority date: 2019-04-09
Filing date: 2019-04-09
Publication date: 2019-08-23
Anticipated expiration: 2039-04-09
Also published as: CN110163248B

Abstract

This application involves a kind of method for visualizing of model evaluation, device, computer equipment and storage mediums, prediction result of the computer using the markup information of test sample collection to unmanned vehicle deep learning model for test sample collection carries out model evaluation, obtains at least one model evaluation result；Wherein, markup information is used to describe the scene information of test sample collection；According to trigger action of the user on interface, show the corresponding model evaluation of trigger action as a result, and/or, process data when model evaluation.Improvement effect to unmanned vehicle deep learning model can be promoted using the above method；Further, computer equipment can according to trigger action of the user on interface, show corresponding model evaluation as a result, and/or, process data when model evaluation improves the working efficiency of model development personnel.

Description

Method for visualizing, device, computer equipment and the storage medium of model evaluation

Technical field

This application involves depth learning technology fields, more particularly to the method for visualizing, device, meter of a kind of model evaluation Calculate machine equipment and storage medium.

Background technique

In the deep learning model development process of unmanned vehicle, model evaluation occupies important a part.By to depth Degree learning model is assessed, and developer can obtain more information from assessment data, helps further to improve The deep learning model of unmanned vehicle.

Existing model evaluation method, to the test result of test data set, obtains confusion matrix according to deep learning model With the model evaluations data such as PR curve, erroneous judgement data that test data is concentrated then are determined according to above-mentioned model evaluation data, Then analysis and assessment are carried out to above-mentioned erroneous judgement data；Further, developer transfers above-mentioned analysis and assessment as a result, coming to depth Learning model improves.

But when after being assessed using the above method deep learning model, model development personnel need from multiple The analysis and assessment of needs are inquired in file as a result, complicated for operation, cause working efficiency low.

Summary of the invention

Based on this, it is necessary in view of the above technical problems, provide the method for visualizing, device, calculating of a kind of model evaluation Machine equipment and storage medium.

A kind of method for visualizing of model evaluation, comprising:

The prediction result of test sample collection is directed to unmanned vehicle deep learning model using the markup information of test sample collection Model evaluation is carried out, at least one model evaluation result is obtained；Wherein, markup information is used to describe the scene letter of test sample collection Breath；

According to trigger action of the user on interface, show the corresponding model evaluation of trigger action as a result, and/or, model Process data when assessment.

The above-mentioned markup information using test sample collection is to unmanned vehicle deep learning model needle in one of the embodiments, Model evaluation is carried out to the prediction result of test sample collection, obtains at least one model evaluation result, comprising:

Unmanned vehicle deep learning model is obtained to the prediction result of test sample collection, and unmanned vehicle is calculated according to prediction result The model evaluation data of deep learning model；

Statistics characteristic analysis is carried out to model evaluation data based on markup information, obtains model evaluation result.

In one of the embodiments, process data when model evaluation include model evaluation data, markup information and At least one of test sample collection.

The above-mentioned trigger action according to user on interface in one of the embodiments, display trigger action are corresponding Model evaluation as a result, and/or, process data when model evaluation, comprising:

Obtain trigger action of the user in assessment task list；Assessment task in the assessment task list corresponds to root Process data when according to the model evaluation result and/or the model evaluation for executing the model evaluation acquisition；

Show process data when the corresponding model evaluation result of assessment task and/or model evaluation being triggered.

Above-mentioned model evaluation data include confusion matrix in one of the embodiments,；Each cell of confusion matrix Data test sample association corresponding with cell data；And each cell data of confusion matrix be based on cell data Process data is associated with when the model evaluation result and/or model evaluation of acquisition；The above-mentioned trigger action according to user on interface, Show the corresponding model evaluation of trigger action as a result, and/or, process data when model evaluation includes:

Obtain trigger action of the user on the cell of the confusion matrix of current presentation；

In response to trigger action, the model evaluation of display and cell data correlation as a result, and/or, with cell data Process data when associated model evaluation.

Process data when model evaluation with cell data correlation includes based on mark in one of the embodiments, The statistical analysis table that information obtains when carrying out statistics characteristic analysis to cell data；Statisticalling analyze table includes markup information Each value correspond to sample size；The process data when model evaluation of above-mentioned display and cell data correlation, further includes:

Obtain the value for the markup information that user chooses in statistical analysis table；

In the test sample of Cell display data correlation, target detection sample corresponding with the value for the markup information chosen This.

In one of the embodiments, in the test sample of above-mentioned Cell display data correlation, believe with the mark chosen Before the corresponding target detection sample of the value of breath, further includes:

Merge the corresponding target detection sample of value for the markup information respectively chosen；

If in the target detection sample after merging, there are identical test samples, retain one of test sample.

It is above-mentioned in one of the embodiments, that statistics characteristic analysis is carried out to model evaluation data based on markup information, it obtains Obtain model evaluation result, comprising:

Test errors sample set is selected from test sample concentration according to model evaluation data；Wherein, test errors sample Collection includes the corresponding test sample of error prediction result；

The value for being based respectively on markup information carries out statistics characteristic analysis to test sample collection and test errors sample set, really Cover half type assessment result.

The above-mentioned value for being based respectively on markup information is to test sample collection and test errors sample in one of the embodiments, This collection carries out statistics characteristic analysis, comprising:

Value based on markup information calculates test sample and the corresponding test sample of each value is concentrated to account for test sample collection The first ratio；

Value based on markup information, calculating the corresponding test sample of each value in test errors sample set, to account for test wrong Accidentally the second ratio of sample set；

According to the first ratio and the second ratio, the significance of each value of markup information is calculated, wherein significance is used for Influence degree of the value of characterization markup information to the prediction result of unmanned vehicle deep learning model.

Above-mentioned markup information includes the acquisition time of test sample collection, collecting test sample in one of the embodiments, At least one of the location information of Weather information and collecting test sample set when collection information.

In one of the embodiments, when markup information includes the location information of collecting test sample set, it is based respectively on The value of markup information carries out test sample collection and test errors sample set before statistics characteristic analysis, further includes:

According to each location information that test sample is concentrated, adjacent location information is subjected to clustering processing, is obtained at least One cluster coordinate；

At least one cluster coordinate is determined as to the value of markup information, wherein a cluster coordinate pair answers markup information A value.

Map is travelled based on preset unmanned vehicle, determines each cluster coordinate corresponding at least one in unmanned vehicle traveling map A path；

At least one path is determined as to the value of markup information；Wherein, a path correspond to markup information one takes Value.

A kind of visualization device of model evaluation, comprising:

Evaluation module is directed to test sample to unmanned vehicle deep learning model for the markup information using test sample collection The prediction result of collection carries out model evaluation, obtains at least one model evaluation result；Wherein, markup information is for describing test specimens The scene information of this collection；

Display module shows the corresponding model evaluation knot of trigger action for the trigger action according to user on interface Fruit, and/or, process data when model evaluation.

A kind of computer equipment, including memory and processor, memory are stored with computer program, processor executes The step of method for visualizing of above-mentioned model evaluation is realized when computer program.

A kind of computer readable storage medium is stored thereon with computer program, when computer program is executed by processor The step of realizing the method for visualizing of above-mentioned model evaluation.

Method for visualizing, device, computer equipment and the storage medium of above-mentioned model evaluation, computer use test sample Prediction result of the markup information of collection to unmanned vehicle deep learning model for test sample collection carries out model evaluation, obtains at least One model evaluation result；Wherein, markup information is used to describe the scene information of test sample collection；According to user on interface Trigger action, the corresponding model evaluation of display trigger action as a result, and/or, process data when model evaluation.Due to computer Prediction result of the equipment using the markup information of test sample collection to unmanned vehicle deep learning model for test sample collection carries out Model evaluation can analyze influence of the different labeled information of test sample to unmanned vehicle deep learning model, to make model Developer can improve model with the concrete scene of binding test sample, improve to unmanned vehicle deep learning model Improvement effect；Further, computer equipment can show corresponding model evaluation according to trigger action of the user on interface As a result, and/or, process data when model evaluation allows model development personnel to pass through display interface to select to need to look into The information such as the model evaluation result looked for, improve working efficiency.

Detailed description of the invention

Fig. 1 is the applied environment figure of the method for visualizing of model evaluation in one embodiment；

Fig. 2 is the flow diagram of the method for visualizing of model evaluation in one embodiment；

Fig. 2A is the schematic diagram of model evaluation data in one embodiment；

Fig. 3 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Fig. 4 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Fig. 5 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Fig. 5 A is the schematic diagram of statistics characteristic analysis process in one embodiment；

Fig. 6 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Fig. 7 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Fig. 7 A is the schematic diagram of statistics characteristic analysis process in one embodiment；

Fig. 8 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Fig. 9 is the flow diagram of the method for visualizing of model evaluation in another embodiment；

Figure 10 is the structural block diagram of the visualization device of model evaluation in one embodiment；

Figure 11 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 12 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 13 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 14 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 15 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 16 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 17 is the structural block diagram of the visualization device of model evaluation in another embodiment；

Figure 18 is the internal structure chart of computer equipment in one embodiment.

Specific embodiment

It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, not For limiting the application.

The method for visualizing of model evaluation provided by the present application can be applied in application environment as shown in Figure 1.Its In, after unmanned vehicle 100 acquires training sample set 110, by unmanned vehicle deep learning model 120 to above-mentioned training sample set 110 It is analyzed and processed, obtains prediction result 130；Computer equipment 140 can be according to above-mentioned prediction result 130 to unmanned vehicle depth Learning model 120 is assessed, and the information such as display model assessment result.Above-mentioned computer equipment 140 can be, but not limited to Various personal computers, laptop, smart phone and tablet computer etc..

In one embodiment, it as shown in Fig. 2, providing a kind of method for visualizing of model evaluation, applies in this way It is illustrated for computer equipment in Fig. 1, comprising:

S101, the prediction for being directed to test sample collection to unmanned vehicle deep learning model using the markup information of test sample collection As a result model evaluation is carried out, at least one model evaluation result is obtained；Wherein, markup information is used to describe the field of test sample collection Scape information.

Wherein, above-mentioned unmanned vehicle deep learning model can be neural network model, be also possible to convolutional network model, right In deep learning model type it is not limited here.Above-mentioned test sample collection can be the set of two-dimension picture, be also possible to The set of 3-D image, such as the set of point cloud data, for above-mentioned test sample collection type it is not limited here.For example, Unmanned vehicle on the way travels a large amount of two dimensional image of acquisition, forms test sample collection, above-mentioned test sample concentration may include nothing The image that people's vehicle is acquired in automatic Pilot is also possible to the image acquired in the state of manned；Computer equipment will In above-mentioned image input unmanned vehicle deep learning model, test sample collection is analyzed by deep learning model and is predicted As a result.Above-mentioned prediction result can be identification road in pedestrian position, can also identify the traffic lights in image color or Person is directed toward, such as left-hand rotation lamp or right-hand rotation lamp, when can also identify red light in the display lamp duration of traffic lights, such as identification picture A length of 20 seconds；Above-mentioned prediction result can be indicated by bounding box (bounding box), can also be indicated by confidence level, right In above-mentioned prediction result type it is not limited here.

Above-mentioned markup information is used to describe the scene information of test sample collection, can be use when acquiring above-mentioned test sample Camera parameter, be also possible to unmanned vehicle speed etc. at the form of acquiring above-mentioned sample, can also be test sample collection Picture or point Yun Zhiliang and picture or point cloud can recognize degree etc., for above-mentioned markup information type it is not limited here. Optionally, markup information can also include test sample collection acquisition time, collecting test sample set when Weather information and At least one of the location information of collecting test sample set information.

Specifically, computer equipment is directed to unmanned vehicle deep learning model in the markup information using test sample collection and surveys When trying the prediction result progress model evaluation of sample set, different markup informations can be analyzed to nothing by data analysis algorithm The prediction result of people's vehicle deep learning model output has any influence；Optionally, computer equipment can also be by obtaining nobody Vehicle deep learning model calculates according to prediction result the mould of unmanned vehicle deep learning model to the prediction result of test sample collection Type assesses data；Then, statistics characteristic analysis is carried out to model evaluation data based on markup information, obtains model evaluation result.

Wherein, above-mentioned model evaluation data refer to the parameter for being able to reflect the predictive ability of unmanned vehicle deep learning model, Above-mentioned model evaluation data can be confusion matrix, be also possible to P-R curve etc., for model evaluation data type herein not It limits.Wherein, confusion matrix is for measuring model accuracy, is mainly used for the true of comparison model prediction result and test sample Real information, every a line in matrix represents the prediction result of test sample collection, each to arrange the real information for representing test sample collection, Cell data in matrix are the test sample quantity of different type of prediction；With the mixed of traffic lights prediction result shown in Fig. 2A Confuse for matrix, the cell data in the third line first row in confusion matrix are 10, and the row where the cell indicates pre- Survey result is amber light, and the column at place indicate that real information is red light, then unmanned vehicle deep learning model is by red light sample predictions Sample size for amber light is 10.Above-mentioned P-R curve is used to indicate the relationship between accurate rate P and recall rate R, wherein accurate Rate P indicates that the ratio of correctly predicted positive sample quantity and test sample quantity, recall rate R indicate correctly predicted positive sample number The ratio of amount and practical positive sample quantity；Such as unmanned vehicle deep learning model owns when predicting the green light in test sample Test sample comprising green light is positive sample, and other test samples are negative sample；The sum of test sample is 100, includes green light Positive sample quantity be 30 when, if having in the corresponding sample of 25 predicted green light that unmanned vehicle deep learning model obtains 20 samples are positive sample, then accurate rate is 20/25=80%, recall rate 20/30=67%.

Specifically, computer equipment can will test after obtaining the prediction result that unmanned vehicle deep learning model obtains The prediction result of sample set and the real information of test sample compare, to obtain above-mentioned confusion matrix；Computer equipment is also Cell data in above-mentioned confusion matrix can be handled, such as further obtain accurate rate and recall rate etc., it obtains P-R curve etc.；The calculation of above-mentioned model evaluation data can be determined according to the type of model evaluation data, herein Without limitation.

Further, computer equipment, can when carrying out statistical characteristic analysis to model evaluation data based on markup information To be analyzed based on the markup information of one of type, can also be analyzed based on the markup information of multiple types；In addition, meter The markup information of a type can also be based respectively on to analyze by calculating machine equipment, then continue to count to above-mentioned analysis result Signature analysis；It does not limit this.For example, computer equipment can be based on collecting test sample set on daytime come analysis model assessment Data；The test sample collection that can also be acquired based on daytime and when weather is fine day.Computer equipment is to model evaluation data When carrying out statistics characteristic analysis, data science tool can be used to analyze, above-mentioned data science tool can be pandas frame Frame is also possible to R frame, for above-mentioned statistics characteristic analysis specific method it is not limited here.

On the basis of above-mentioned steps, computer equipment can obtain model evaluation as a result, above-mentioned model evaluation result can To be a kind of markup information, have for the hints model developer markup information to the accuracy of unmanned vehicle deep learning model Large effect；It is existing to be also possible to the mistake that the markup information of above-mentioned influence unmanned vehicle deep learning model accuracy may cause As, such as " unmanned vehicle deep learning model is easy the amber light test sample that night acquires being predicted as green light "；For above-mentioned mould The concrete form of type assessment result is it is not limited here.

S102, the trigger action according to user on interface, the corresponding model evaluation of display trigger action as a result, and/or, Process data when model evaluation.

Computer equipment can be interacted by interface and user, will when user executes trigger action on interface Model evaluation corresponding with the trigger action as a result, and/or, process data when model evaluation is shown.Wherein, above-mentioned boundary Face can be webpage, be also possible to the interface as the result is shown in model evaluation tool, and the type at above-mentioned interface is not done herein It limits.Above-mentioned trigger action can be clicking operation, can also be through drag operation, for above-mentioned trigger action type herein Without limitation.User when carrying out trigger action on interface, touch when can be to each model evaluation by corresponding assessment task Hair operation can also carry out trigger action to different training sample sets, it is not limited here.

Above-mentioned trigger action can correspond to all model evaluations of above-mentioned steps acquisition as a result, and/or, when model evaluation Process data, be also possible to the different location triggered according to user, corresponding different model evaluation as a result, and/or, model is commented Process data when estimating.

Wherein, process data when above-mentioned model evaluation may include computer equipment when carrying out model evaluation to prediction As a result analytic process, such as the data statistics process that can be presented in tabular form；Optionally, number of passes is crossed when model evaluation According to may include at least one of model evaluation data, markup information and test sample collection.

Specifically, computer equipment show above-mentioned model evaluation as a result, and/or, when process data when model evaluation, The corresponding all results of above-mentioned trigger action can all be shown, can also by the display mode of different levels, after It is continuous to be interacted with user, so that user is selected desired model evaluation as a result, and/or, process data when model evaluation；It is right In above-mentioned display mode it is not limited here.Computer equipment can pass through jump page after user executes trigger action Show corresponding model evaluation as a result, and/or, process data when model evaluation, can also by way of pop-up window, or The modes of person's suspension windows is shown, is not limited this.

The method for visualizing of above-mentioned model evaluation, computer is using the markup information of test sample collection to unmanned vehicle depth The prediction result that model is practised for test sample collection carries out model evaluation, obtains at least one model evaluation result；Wherein, it marks Information is used to describe the scene information of test sample collection；According to trigger action of the user on interface, show that trigger action is corresponding Model evaluation as a result, and/or, process data when model evaluation.Since computer equipment uses the mark of test sample collection Prediction result of the information to unmanned vehicle deep learning model for test sample collection carries out model evaluation, can analyze test sample Influence of the different labeled information to unmanned vehicle deep learning model, to make model development personnel can be with binding test sample Concrete scene improves model, improves the improvement effect to unmanned vehicle deep learning model；Further, computer is set It is standby can according to trigger action of the user on interface, show corresponding model evaluation as a result, and/or, mistake when model evaluation Number of passes evidence allows model development personnel to pass through display interface to select the information such as the model evaluation result required to look up, mentions Working efficiency is risen.

Fig. 3 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to calculating Machine equipment show the corresponding model evaluation of trigger action as a result, and/or, a kind of specific side of process data when model evaluation Formula, on the basis of the above embodiments, as shown in figure 3, above-mentioned S102 includes:

S201, trigger action of the user in assessment task list is obtained；The assessment task assessed in task list is corresponding Process data when according to the model evaluation result and/or model evaluation for executing model evaluation acquisition.

Wherein, multiple assessment tasks can be corresponded in above-mentioned assessment task list, computer equipment is executing a model After assessment, can ought time model evaluation model evaluation obtained as a result, and/or, execute a process when model evaluation Data are stored according to assessment task；In addition, when above-mentioned assessment task can also include that computer equipment executes model evaluation The data such as parameter setting, it is not limited here for corresponding data type in above-mentioned assessment task.

Specifically, above-mentioned assessment task can be shown on interface by way of table, can be shown in the form of icon Show on above-mentioned interface；Show that above-mentioned execution task corresponding model evaluation time, assessment task are responsible for furthermore it is also possible to combine The start context of the information such as people and corresponding unmanned vehicle deep learning model, for the display mode of above-mentioned assessment task, This is without limitation.

User, can be by double-clicking the assessment task in the assessment task for selecting to need from above-mentioned assessment task list A line in corresponding table selects, and can also be selected by the tick boxes carried in selection assessment task list, for The triggering mode of above-mentioned assessment task is it is not limited here.

Process data when the corresponding model evaluation result of assessment task and/or model evaluation that S202, display are triggered.

Computer equipment can determine commenting for user's needs after obtaining above-mentioned trigger action operation according to trigger action Estimate task, then by the corresponding model evaluation result of above-mentioned assessment task and/or model evaluation when process data show. Similar with the description in above-mentioned S102 for above-mentioned display mode, details are not described herein.

The method for visualizing of above-mentioned model evaluation, computer equipment can select to need by user from assessment task list The assessment task wanted, to show process data when the corresponding model evaluation result of the assessment task and/or model evaluation, so that User does not need to go to search from the file of computer equipment, improves the working efficiency of user.

Fig. 4 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to calculating Machine equipment show the corresponding model evaluation of trigger action as a result, and/or, the specific side of another kind of process data when model evaluation Formula, on the basis of the above embodiments, as shown in figure 4, above-mentioned S102 includes:

S301, trigger action of the user on the cell of the confusion matrix of current presentation is obtained.

Wherein, the model evaluation data that computer equipment obtains include confusion matrix, each cell number of confusion matrix It is associated with according to test sample corresponding with cell data；And with the model evaluation result obtained based on cell data and/or Process data is associated with when model evaluation.Specifically, computer equipment based on markup information to the cell data of confusion matrix into After row statistics characteristic analysis, the model evaluation result of acquisition and said units lattice data can be associated, determination unit lattice The incidence relation of data and model evaluation result；In addition, computer equipment can also by model evaluation process data with it is upper State each unit lattice to be associated, the incidence relation of determination process data and cell data, for example, each cell data with The corresponding test sample association of the cell data.

For the confusion matrix for continuing the traffic lights prediction result shown in Fig. 2A,；Computer equipment to the first row three It, can be by three lists of the model evaluation result A of acquisition and above-mentioned the first row after a cell data carry out statistics characteristic analysis First lattice data are associated respectively；In addition, computer equipment can also carry out statistics spy to tertial three cell data Sign analysis, and the model evaluation result B of acquisition is associated with three cell data of above-mentioned first row respectively respectively；On It states the same cell data and can be associated with different model evaluations as a result, different cell data or being associated with same Model evaluation is as a result, it is not limited here.Computer equipment is by each unit of model evaluation result and above-mentioned confusion matrix After lattice are associated, above-mentioned confusion matrix can be shown on interface, and each cell data of confusion matrix are added Add a response, such as sets a virtual control for cell；After user triggers said units lattice data, computer The available above-mentioned trigger action of equipment, determine user need check be which cell data correlation model evaluation knot The information such as fruit.

S302, in response to trigger action, the model evaluation of display and cell data correlation as a result, and/or, with cell The process data when model evaluation of data correlation.

Further, computer equipment can respond above-mentioned trigger action, determine that user selects according to above-mentioned trigger action The cell data selected are then based on the incidence relation of said units lattice data Yu model evaluation data, and/or, said units The incidence relation of lattice data and process data, by user selection cell data correlation model evaluation as a result, and/or, with The process data when model evaluation of cell data correlation is shown.

The method for visualizing of above-mentioned model evaluation, computer equipment are carried out by the cell data of confusion matrix and user Interaction, by obtain user selection cell data, by the model evaluation with cell data correlation as a result, and/or, with list The process data when model evaluation of first lattice data correlation is shown, user is directly selected according to cell data The information such as the model evaluation result for needing to check, further promote working efficiency.

Fig. 5 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to calculating Machine equipment show the corresponding model evaluation of trigger action as a result, and/or, the specific side of another kind of process data when model evaluation Formula, on the basis of the above embodiments, as shown in figure 5, above-mentioned S302 includes:

S401, the value for obtaining the markup information that user chooses in statistical analysis table.

For user after the cell data by confusion matrix, computer equipment can be by said units lattice data correlation Process data show, wherein process data when the above-mentioned model evaluation with cell data correlation may include base The statistical analysis table obtained when markup information carries out statistics characteristic analysis to cell data；Statisticalling analyze table includes mark Each value of note information corresponds to sample size.For the confusion matrix for continuing the traffic lights prediction result shown in Fig. 2A, on Stating statistical analysis table can be as shown in Figure 5 A, each value and the corresponding sample number of each value including markup information Amount, there are also the data such as percentage obtained in statistical analysis process.

User can select the corresponding model of value for needing the markup information checked to comment by above-mentioned statistical analysis table Estimate the information such as result, can choose the value of a markup information, it can also be with the value of the multiple markup informations of simultaneous selection, herein Without limitation.Computer equipment can obtain the value for each markup information that user chooses according to the trigger action of user.

S402, Cell display data correlation test sample in, target corresponding with the value for the markup information chosen Test sample.

Specifically, computer equipment is after obtaining the value of markup information that user chooses, can from the cell In associated test sample, target detection sample corresponding with the value of above-mentioned markup information in extraction.For example, user's selection It is amber light for real information, prediction result is the cell data of green light, and computer equipment shows the statistical analysis in Fig. 5 A Table；It is selected in test errors sample set in user by statistical analysis table, when the value of markup information is evening, computer Equipment can be by each test sample of the unit associated 30, and acquisition time is that 20 each test samples in evening are determined as target survey Sample sheet.Computer equipment can be corresponding with the value for the markup information chosen by the test sample of cell data correlation Test sample is determined as target detection sample, can also be by a test sample in above-mentioned test sample, or a part Test sample is determined as target detection sample, it is not limited here.

Further, user can choose the value of multiple markup informations simultaneously, the value with above-mentioned multiple markup informations The quantity of corresponding test sample may be bigger, thereby increases and it is possible to the case where there are test sample repetitions；Computer equipment is being shown Before above-mentioned target detection sample, target detection sample can be handled, taking for the markup information respectively chosen can be merged It is worth corresponding target detection sample；If retaining wherein one there are identical test sample in the target detection sample after merging A test sample.For example, the value for two markup informations that computer equipment can choose, one of them is daytime, another For fine day；So computer equipment can obtain the corresponding one group of target detection sample of value of above-mentioned two markup information respectively This, then merges above-mentioned two groups of target detection samples；Due in the target detection sample after above-mentioned merging, it is understood that there may be Two identical target detection samples, then computer equipment can delete one of target detection sample.

Computer equipment can be shown above-mentioned on the basis of obtaining above-mentioned target detection sample near confusion matrix Target detection sample, such as picture is suspended into display on the right side of display interface；It can also be by by above-mentioned target detection sample Centralized displaying in another display window is extracted, it is not limited here for above-mentioned display method.

The method for visualizing of above-mentioned model evaluation, computer equipment can be such that model opens by displaying target test sample Hair personnel are intuitive to see the feature of the test sample of prediction error, the sample of prediction error and prediction can also correctly be surveyed Sample originally compares, and is conducive to model development personnel's improved model.

Fig. 6 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to calculating Machine equipment carries out a kind of concrete mode of statistics characteristic analysis based on markup information to model evaluation data, in above-described embodiment On the basis of, as shown in fig. 6, above-mentioned carry out statistics characteristic analysis to the model evaluation data based on the markup information, obtain Model evaluation result, comprising:

S501, test errors sample set is selected from test sample concentration according to model evaluation data；Wherein, test errors Sample set includes the corresponding test sample of error prediction result.

Specifically, computer equipment can be according to model evaluation data, to unmanned vehicle deep learning model prediction mistake Test sample is analyzed, and test errors sample set is selected；Above-mentioned test sample collection can be the test sample and concentrate institute The corresponding sample of wrong prediction result, can also select a part of error prediction structure pair according to above-mentioned model evaluation data The test sample answered；The selection mode of above-mentioned test errors sample set is it is not limited here.

Above-mentioned test errors sample set can be the sample set selected according to model evaluation data, can also be basis Multiple sample sets that multiple data in model evaluation data choose, for above-mentioned test errors sample set type herein Without limitation.

4 units for the confusion matrix for continuing the traffic lights prediction result shown in Fig. 2A, in above-mentioned confusion matrix Lattice data correspond to error prediction as a result, computer equipment can select test errors sample according to above-mentioned each unit lattice data Collection, for example, computer equipment can choose the corresponding test specimens of the maximum cell data of numerical value in above-mentioned 4 cell data This is test errors sample set, is green light for prediction result and 40 test samples that real information is red light are test errors sample This collection；In addition, it is amber light that computer equipment, which also can choose real information, and prediction result is two of red light and green light Cell data 20 and 30 corresponding test samples are test errors sample set, and computer equipment can test above-mentioned 50 Sample is determined as a test errors sample set, by real information can also be red light and prediction result is 20 tests of red light Sample and real information are amber light and 30 test samples that prediction result is green light are identified as a test errors sample Collection, that is to say, that computer equipment selects two test errors sample sets.

S502, the value for being based respectively on markup information carry out statistical nature point to test sample collection and test errors sample set Analysis, determines model evaluation result.

Wherein, the value of above-mentioned markup information refers to the different scenes information for including in the type of markup information.For example, mark When to infuse information be acquisition time, the value of markup information can be specific acquisition moment, such as eight o'clock sharps, can also with daytime or Person is at night；When markup information is Weather information, the value of markup information may include fine day, rainy day, be also possible to haze index； When markup information is location information, the value of markup information can be coordinate position when collecting test sample, location information Value is also possible to the information obtained after handling above-mentioned coordinate position, such as can be city, is also possible to link name Claim etc..

In addition, the value of markup information may include a type of markup information, such as the value of markup information is white It and at night；The value of markup information also may include a plurality of types of markup informations, such as markup information type includes acquisition When time and Weather information, the value of markup information can be fine day on daytime-, cloudy day on daytime-, evening-fine day and evening-yin It；For above-mentioned markup information value form it is not limited here.Computer equipment can analyze acquisition on daytime and test Sample model evaluation data corresponding with the test sample of evening acquisition, to determine whether acquisition time difference to unmanned vehicle depth The accuracy of learning model has an impact, and specific statistical analysis technique is similar with above-mentioned S102, it is not limited here.

Above-mentioned model analysis method, value of the computer equipment based on markup information, respectively to test sample collection and test Error sample collection carries out statistics characteristic analysis, is conducive to computer equipment and obtains test errors sample set in different markup informations Value in terms of statistical nature, determine model evaluation as a result, promoting the improvement effect to deep learning model in turn.

Fig. 7 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to calculating The value that machine equipment is based respectively on markup information carries out the tool of statistics characteristic analysis to test sample collection and test errors sample set Body mode, on the basis of the above embodiments, as shown in fig. 7, above-mentioned S502 includes:

S601, the value based on markup information calculate test sample and the corresponding test sample of each value are concentrated to account for test First ratio of sample set.

S602, the value based on markup information calculate the corresponding test sample of each value in test errors sample set and account for Second ratio of test errors sample set.

S603, according to the first ratio and the second ratio, calculate the significance of each value of markup information.

Specifically, computer equipment can calculate the corresponding test sample of each value based on the value of markup information It accounts for the corresponding test sample of each value in the first ratio and test errors sample set of test sample collection and accounts for my wrong sample of test Second ratio of this collection.

For the confusion matrix for continuing the traffic lights prediction result shown in Fig. 2A, in the test specimens that real information is amber light For this collection, the quantity of test sample collection is 1000, wherein the acquisition time of 720 test samples is daytime, 280 test specimens This acquisition time is at night；It for 30 test samples of green light is test errors sample that computer equipment, which selects prediction result, Collection, in above-mentioned test errors sample set, having the acquisition time of 10 test samples is daytime, there is the acquisition of 20 test samples Time is at night；Computer carries out statistics characteristic analysis processing, available above-mentioned first ratio to above-mentioned model evaluation data Are as follows: the test sample accounting on daytime is 72%, and the test sample accounting in evening is 28%；Available above-mentioned second ratio simultaneously For in test errors sample set, the test sample accounting on daytime is 33%, and the test sample accounting in evening is 67%, above-mentioned Analytic process is shown in Fig. 7 A.

Computer equipment on the basis of obtaining above-mentioned first ratio and the second ratio, can according to above-mentioned first ratio and Second ratio calculates the significance of each value of markup information.Wherein, significance is used to characterize the value of markup information to nothing The influence degree of the prediction result of people's vehicle deep learning model.Above-mentioned significance can be above-mentioned markup information value it is different and The probability for causing test sample collection different with the second ratio with above-mentioned first ratio in error checking sample set.Computer equipment Above-mentioned significance can be obtained by the data science tool in above-mentioned S102.

For the confusion matrix for continuing the traffic lights prediction result shown in the 2A, by the first ratio and the second ratio into Row analysis, computer equipment is available, and it is acquisition at night that only 28% test sample is concentrated in test sample, and is tested Error sample be concentrated with 67% test sample be at night acquisition, therefore the value of markup information be evening when, to unmanned vehicle The prediction accuracy of deep learning model has a significant impact, and unmanned vehicle deep learning model is easy to test the amber light of evening acquisition Sample predictions are green light.

Further, computer equipment, can be according to each of markup information after the significance for obtaining each value The highest value of significance is determined as influencing the factor of unmanned vehicle deep learning model evaluation result by the significance of value.Example It such as, can be at night, to be determined as the factor of unmanned vehicle deep learning model evaluation result for the value of markup information.

The method for visualizing of above-mentioned model evaluation, value of the computer equipment based on markup information calculate test sample collection And in test errors sample set, the first ratio and the second ratio shared by the corresponding test sample of each value；It can basis Which value above-mentioned first ratio and the second ratio more accurately determine to the prediction result of unmanned vehicle deep learning model It is affected, and then improves model development personnel to model for the value of above-mentioned markup information, promoted to unmanned vehicle The improvement effect of deep learning model.

Fig. 8 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to marking Information includes the case where the location information of collecting test sample set, on the basis of the above embodiments, as shown in figure 5, above-mentioned Before S502 further include:

Adjacent location information is carried out clustering processing, obtained by S701, each location information concentrated according to test sample At least one cluster coordinate.

Specifically, unmanned vehicle may carry out multi collect in the same position when acquiring above-mentioned test sample collection, such as It can concentrate that there may be plurality of pictures to be in test sample by the image information of the different angle acquisition positions, therefore Same position is collected, adjacent location information can be carried out clustering processing by computer equipment, obtain at least one Cluster coordinate.When computer equipment carries out clustering processing to location information, the distance between different location information threshold can be set Value, when the distance between location information of two test samples is less than above-mentioned distance threshold, computer equipment can consider this The corresponding position of two test samples is identical, then clusters the location information of two test samples, obtains a cluster Coordinate.

Computer equipment, can be above-mentioned multiple surveys when the location information of multiple test samples is carried out clustering processing One of location information in the location information of sample sheet is determined as clustering coordinate, can also be to above-mentioned multiple test samples The coordinate of location information is averaged, and the coordinate after being then averaged is determined as clustering coordinate；For obtaining for above-mentioned cluster coordinate The mode of obtaining is not limited thereto.

S702, the value that at least one cluster coordinate is determined as to markup information, wherein a cluster coordinate pair should mark One value of information.

Specifically, above-mentioned cluster coordinate can be determined as the value of markup information by computer equipment, so that computer is set It is standby to carry out statistics characteristic analysis based on each cluster coordinate pair model evaluation data.For example, computer equipment can determine survey Sample originally concentrates those corresponding test samples of cluster coordinate to be easy by unmanned vehicle deep learning model prediction mistake.

Further, computer equipment can pass through the above-mentioned cluster coordinate of map denotation.

Cluster coordinate is determined as the value of markup information, made by the method for visualizing of above-mentioned model evaluation, computer equipment Obtaining computer equipment can be easy according to the test sample of above-mentioned which position of cluster coordinate analysis by unmanned vehicle deep learning mould Type prediction error, the test sample for allowing model development personnel to resurvey the position are trained model, Lifting Modules The accuracy of type prediction.

Fig. 9 is the flow diagram of the method for visualizing of model evaluation in another embodiment；The present embodiment is related to marking Another method when information includes the location information of collecting test sample set, on the basis of the above embodiments, such as Fig. 9 institute Show, before above-mentioned S502 further include:

Adjacent location information is carried out clustering processing, obtained by S801, each location information concentrated according to test sample At least one cluster coordinate.

Specifically, the acquisition pattern of above-mentioned cluster coordinate is similar with the description in above-mentioned S401, and details are not described herein.

S802, map is travelled based on preset unmanned vehicle, determines that each cluster coordinate is corresponding in unmanned vehicle traveling map At least one path.

Computer equipment can be based on preset unmanned vehicle form map, cluster is sat after obtaining above-mentioned cluster coordinate Mark and corresponding coordinate points in above-mentioned unmanned vehicle traveling map are corresponding, determine that above-mentioned cluster coordinate is to belong in map That path；The path that computer equipment is determined can be the number in path, can also be the coordinate range in path, can be with It is the title in path, it is not limited here.

S803, the value that at least one path is determined as to markup information；Wherein, a path corresponds to the one of markup information A value.

Above-mentioned path can be determined as the value of markup information by computer equipment, computer equipment is based on each Path carries out statistics characteristic analysis to model evaluation data.For example, computer equipment can determine that test sample concentrates those roads The corresponding test sample of diameter is easy by unmanned vehicle deep learning model prediction mistake.

Further, computer equipment, can after carrying out statistics characteristic analysis to model evaluation data based on each path The highest path of significance to be determined as to influencing the destination path of unmanned vehicle deep learning model evaluation result, and it is based on nobody Vehicle travels map, is route to be optimized by the route determination comprising destination path.Further, computer equipment can also will be upper It states path to be shown on map, the route comprising destination path can also be shown, and by the way that different color areas is arranged Divide different routes.

For the confusion matrix for continuing the traffic lights prediction result shown in the 2A, computer equipment is to each of test sample collection A location information carries out clustering processing, is then based on preset unmanned vehicle traveling map and determines the corresponding path of cluster coordinate, " road Ke Yun " " Zhongshan Road " " five hill paths " " Beijing Road " is determined as to the value of markup information, based on above-mentioned each path to obscuring square Cell data in battle array carry out statistics characteristic analysis, and the corresponding test sample in available each path accounts for test sample collection The second ratio that the corresponding test sample of first ratio and each path accounts for test errors sample set is being surveyed as shown in Figure 9 A This concentration of sample, the ratio that " road Ke Yun " corresponding test sample accounts for test sample collection is 10%, and in test errors sample set In, the ratio that " road Ke Yun " corresponding test sample accounts for test errors sample set is 40%, that is to say, that unmanned vehicle deep learning Model is easy the test sample prediction error that will be acquired in " road Ke Yun ", which can be determined as influencing unmanned vehicle depth Practise the destination path of model evaluation result；Further, computer equipment can will comprising " road Ke Yun " route determination be to Optimize route.

Path is determined as the value of markup information by the method for visualizing of above-mentioned model evaluation, computer equipment, makes to succeed in one's scheme Calculating machine equipment can be easy by unmanned vehicle deep learning model prediction mistake according to the test sample in which path of above-mentioned path analysis Accidentally, the test sample for allowing model development personnel to resurvey the path is trained model, lift scheme prediction Accuracy, simultaneous computer equipment can make unmanned vehicle exist by that will be route to be optimized comprising the route determination of destination path When programme path, above-mentioned destination path is avoided with there can be deviation.

It should be understood that although each step in the flow chart of Fig. 2-9 is successively shown according to the instruction of arrow, These steps are not that the inevitable sequence according to arrow instruction successively executes.Unless expressly stating otherwise herein, these steps Execution there is no stringent sequences to limit, these steps can execute in other order.Moreover, at least one in Fig. 2-9 Part steps may include that perhaps these sub-steps of multiple stages or stage are not necessarily in synchronization to multiple sub-steps Completion is executed, but can be executed at different times, the execution sequence in these sub-steps or stage is also not necessarily successively It carries out, but can be at least part of the sub-step or stage of other steps or other steps in turn or alternately It executes.

In one embodiment, as shown in Figure 10, a kind of visualization device of model evaluation is provided, comprising: assessment mould Block 10 and display module 20, in which:

Evaluation module 10 is directed to test specimens to unmanned vehicle deep learning model for the markup information using test sample collection The prediction result of this collection carries out model evaluation, obtains at least one model evaluation result；Wherein, markup information is for describing test The scene information of sample set.

Display module 20 shows the corresponding model evaluation of trigger action for the trigger action according to user on interface As a result, and/or, process data when model evaluation.

Above method embodiment may be implemented in the visualization device of model evaluation provided by the embodiments of the present application, realizes Principle is similar with technical effect, and details are not described herein.

In one embodiment, as shown in figure 11, on the basis of the above embodiments, above-mentioned evaluation module 10 includes:

Computational submodule 101, for obtaining unmanned vehicle deep learning model to the prediction result of test sample collection, and according to The model evaluation data of prediction result calculating unmanned vehicle deep learning model.

Statistic submodule 102 obtains model for carrying out statistics characteristic analysis to model evaluation data based on markup information Assessment result.

In one embodiment, process data when model evaluation includes model evaluation data, markup information and test At least one of sample set.

In one embodiment, as shown in figure 12, on the basis of the above embodiments, display module 20 includes:

Submodule 201 is triggered, for obtaining trigger action of the user in assessment task list；It assesses in task list Process data when assessment task is corresponded to according to the model evaluation result and/or model evaluation for executing model evaluation acquisition.

Display sub-module 202, for showing the corresponding model evaluation result of assessment task and/or model evaluation that are triggered When process data.

In one embodiment, as shown in figure 12, on the basis of the above embodiments, model evaluation data include obscuring square Battle array；The test sample association corresponding with cell data of each cell data of confusion matrix；And each list of confusion matrix First lattice data are associated with process data when the model evaluation result and/or model evaluation obtained based on cell data；

Submodule 201 is triggered, is also used to obtain trigger action of the user on the cell of the confusion matrix of current presentation.

Display sub-module 202 is also used in response to trigger action, the model evaluation knot of display and cell data correlation Fruit, and/or, process data when model evaluation with cell data correlation.

In one embodiment, as shown in figure 13, on the basis of the above embodiments, with the model of cell data correlation Process data when assessment includes the statistical analysis obtained when carrying out statistics characteristic analysis to cell data based on markup information Table；Statistical analysis table includes that each value of markup information corresponds to sample size；Above-mentioned display sub-module 202 further includes, Further include:

Acquiring unit 2021, for obtaining the value for the markup information that user chooses in statistical analysis table.

Display unit 2022, in the test sample for Cell display data correlation, with taking for the markup information chosen It is worth corresponding target detection sample.

In one embodiment, on the basis of the above embodiments, above-mentioned display unit 222 is also used to: merging is respectively chosen Markup information the corresponding target detection sample of value；If in the target detection sample after merging, there are identical test specimens This, then retain one of test sample.

In one embodiment, as shown in figure 14, on the basis of the above embodiments, above-mentioned statistic submodule 102 includes:

Selecting unit 1021, for selecting test errors sample set from test sample concentration according to model evaluation data； Wherein, test errors sample set includes the corresponding test sample of error prediction result.

Statistic unit 1022, for be based respectively on the value of markup information to test sample collection and test errors sample set into Row statistics characteristic analysis determines model evaluation result.

In one embodiment, as shown in figure 15, on the basis of the above embodiments, above-mentioned statistic unit 1022 includes:

First computation subunit 10221 calculates test sample and concentrates each value pair for the value based on markup information The test sample answered accounts for the first ratio of test sample collection

Second computation subunit 10222 calculates each in test errors sample set take for the value based on markup information It is worth the second ratio that corresponding test sample accounts for test errors sample set.

Third computation subunit 10223, for calculating each value of markup information according to the first ratio and the second ratio Significance, wherein significance is used to characterize the value of markup information to the shadow of the prediction result of unmanned vehicle deep learning model The degree of sound.

In one embodiment, on the basis of the above embodiments, markup information include test sample collection acquisition time, At least one of the location information of Weather information and collecting test sample set when collecting test sample set information.

In one embodiment, as shown in figure 16, when markup information includes the location information of collecting test sample set, On the basis of above-described embodiment, above-mentioned statistic submodule 102 further include:

Cluster cell 1023, each location information for being concentrated according to test sample carry out adjacent location information Clustering processing obtains at least one cluster coordinate；

Determination unit 1024, at least one cluster coordinate to be determined as to the value of markup information, wherein a cluster Coordinate pair answers a value of markup information.

In one embodiment, as shown in figure 16, when markup information includes the location information of collecting test sample set, On the basis of above-described embodiment, determination unit 1024 is also used to travel map based on preset unmanned vehicle, determines each cluster coordinate At least one corresponding path in unmanned vehicle traveling map；At least one path is determined as to the value of markup information；Wherein, One path corresponds to a value of markup information.

In one embodiment, as shown in figure 17, on the basis of the above embodiments, above-mentioned statistic unit 302 further includes It determines subelement 10224, for the significance according to each value of markup information, the highest value of significance is determined as shadow Ring the factor of unmanned vehicle deep learning model evaluation result.

In one embodiment, on the basis of the above embodiments, above-mentioned determining subelement 10224 is specifically used for: will show The highest path of work degree is determined as influencing the destination path of unmanned vehicle deep learning model evaluation result；Travel ground based on unmanned vehicle Route determination comprising destination path is route to be optimized by figure.

The specific of visualization device about model evaluation limits the visualization that may refer to above for model evaluation The restriction of method, details are not described herein.Modules in the visualization device of above-mentioned model evaluation can be fully or partially through Software, hardware and combinations thereof are realized.Above-mentioned each module can be embedded in the form of hardware or independently of the place in computer equipment It manages in device, can also be stored in a software form in the memory in computer equipment, in order to which processor calls execution or more The corresponding operation of modules.

In one embodiment, a kind of computer equipment is provided, which can be terminal, internal structure Figure can be as shown in figure 18.The computer equipment includes the processor connected by system bus, memory, network interface, shows Display screen and input unit.Wherein, the processor of the computer equipment is for providing calculating and control ability.The computer equipment Memory includes non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system and computer Program.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The meter The network interface for calculating machine equipment is used to communicate with external terminal by network connection.When the computer program is executed by processor To realize a kind of method for visualizing of model evaluation.The display screen of the computer equipment can be liquid crystal display or electronic ink Water display screen, the input unit of the computer equipment can be the touch layer covered on display screen, be also possible to computer equipment Key, trace ball or the Trackpad being arranged on shell can also be external keyboard, Trackpad or mouse etc..

It will be understood by those skilled in the art that structure shown in Figure 18, only part relevant to application scheme The block diagram of structure, does not constitute the restriction for the computer equipment being applied thereon to application scheme, and specific computer is set Standby may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.

In one embodiment, a kind of computer equipment, including memory and processor are provided, is stored in memory Computer program, the processor perform the steps of when executing computer program

Computer equipment provided in this embodiment, implementing principle and technical effect are similar with above method embodiment, This is repeated no more.

In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated Machine program performs the steps of when being executed by processor

Computer readable storage medium provided in this embodiment, implementing principle and technical effect and above method embodiment Similar, details are not described herein.

Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is controlled by computer program to complete, the computer program can be stored in a non-volatile computer In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, To any reference of memory, storage, database or other media used in each embodiment provided herein, Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..

Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance Shield all should be considered as described in this specification.

The several embodiments of the application above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously It cannot therefore be construed as limiting the scope of the patent.It should be pointed out that coming for those of ordinary skill in the art It says, without departing from the concept of this application, various modifications and improvements can be made, these belong to the protection of the application Range.Therefore, the scope of protection shall be subject to the appended claims for the application patent.

Claims

1. a kind of method for visualizing of model evaluation, which is characterized in that the described method includes:

The prediction result of the test sample collection is directed to unmanned vehicle deep learning model using the markup information of test sample collection Model evaluation is carried out, at least one model evaluation result is obtained；Wherein, the markup information is for describing the test sample collection Scene information；

According to trigger action of the user on interface, show the corresponding model evaluation of the trigger action as a result, and/or, model Process data when assessment.

2. the method according to claim 1, wherein the markup information using test sample collection is to unmanned vehicle Deep learning model carries out model evaluation for the prediction result of the test sample collection, obtains at least one model evaluation knot Fruit, comprising:

Unmanned vehicle deep learning model is obtained to the prediction result of test sample collection, and the nothing is calculated according to the prediction result The model evaluation data of people's vehicle deep learning model；

Statistics characteristic analysis is carried out to the model evaluation data based on the markup information, obtains model evaluation result.

3. according to the method described in claim 2, it is characterized in that, the process data when model evaluation includes the model Assess at least one of data, the markup information and described test sample collection.

4. method according to claim 1-3, which is characterized in that the triggering behaviour according to user on interface Make, show the corresponding model evaluation of the trigger action as a result, and/or, process data when model evaluation, comprising:

Obtain trigger action of the user in assessment task list；The corresponding basis of assessment task in the assessment task list is held Process data when the model evaluation result and/or the model evaluation that the row model evaluation obtains；

5. according to the method described in claim 2, it is characterized in that, the model evaluation data include confusion matrix；It is described mixed Confuse each cell data corresponding with cell data test sample association of matrix；And each of described confusion matrix Cell data are associated with process data when the model evaluation result and/or model evaluation obtained based on the cell data； The trigger action according to user on interface, show the corresponding model evaluation of the trigger action as a result, and/or, model Process data when assessment includes:

In response to the trigger action, the model evaluation of display and the cell data correlation as a result, and/or, with the list The process data when model evaluation of first lattice data correlation.

6. according to the method described in claim 5, it is characterized in that, mistake when model evaluation with the cell data correlation Number of passes evidence includes the statistical analysis table obtained when carrying out statistics characteristic analysis to the cell data based on the markup information Lattice；The statistical analysis table includes that each value of markup information corresponds to sample size；The display and the cell number Process data when according to associated model evaluation, further includes:

Obtain the value for the markup information that user chooses in the statistical analysis table；

In the test sample for showing the cell data correlation, target corresponding with the value of the markup information chosen is surveyed Sample sheet.

7. according to the method described in claim 6, it is characterized in that, the test sample of the display cell data correlation In, before target detection sample corresponding with the value of the markup information chosen, further includes:

Merge the corresponding target detection sample of value of each markup information chosen；

8. according to the method described in claim 2, it is characterized in that, described be based on the markup information to the model evaluation number According to statistics characteristic analysis is carried out, model evaluation result is obtained, comprising:

Test errors sample set is selected from test sample concentration according to the model evaluation data；Wherein, the test Error sample collection includes the corresponding test sample of error prediction result；

The value for being based respectively on the markup information carries out statistics spy to the test sample collection and the test errors sample set Sign analysis, determines the model evaluation result.

9. according to the method described in claim 8, it is characterized in that, the value for being based respectively on the markup information is to described Test sample collection and the test errors sample set carry out statistics characteristic analysis, comprising:

Based on the value of the markup information, calculates the test sample and the corresponding test sample of each value is concentrated to account for the survey Try the first ratio of sample set；

Based on the value of the markup information, calculates the corresponding test sample of each value in the test errors sample set and account for survey Second ratio of trial and error mistake sample set；

According to first ratio and second ratio, the significance of each value of the markup information is calculated, wherein institute Significance is stated for characterizing the value of markup information to the influence degree of the prediction result of the unmanned vehicle deep learning model.

10. according to the method described in claim 9, it is characterized in that, the markup information includes adopting for the test sample collection Weather information when collecting the time, acquiring the test sample collection and in the location information of the acquisition test sample collection at least A kind of information.

11. according to the method described in claim 10, it is characterized in that, when the markup information includes acquiring the test sample When the location information of collection, the value for being based respectively on the markup information is to the test sample collection and the test errors sample This collection carries out before statistics characteristic analysis, further includes:

According to each location information that the test sample is concentrated, adjacent location information is subjected to clustering processing, is obtained at least One cluster coordinate；

At least one described cluster coordinate is determined as to the value of the markup information, wherein a cluster coordinate pair is answered described One value of markup information.

12. according to the method described in claim 10, it is characterized in that, when the markup information includes acquiring the test sample When the location information of collection, the value for being based respectively on the markup information is to the test sample collection and the test errors sample This collection carries out before statistics characteristic analysis, further includes:

Based on preset unmanned vehicle travel map, determine each cluster coordinate the unmanned vehicle travel map in it is corresponding extremely A few path；

At least one described path is determined as to the value of the markup information；Wherein, a path corresponds to the markup information A value.

13. a kind of visualization device of model evaluation, which is characterized in that described device includes:

Evaluation module is directed to the test sample to unmanned vehicle deep learning model for the markup information using test sample collection The prediction result of collection carries out model evaluation, obtains at least one model evaluation result；Wherein, the markup information is for describing institute State the scene information of test sample collection；

Display module shows the corresponding model evaluation knot of the trigger action for the trigger action according to user on interface Fruit, and/or, process data when model evaluation.

14. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists In the step of processor realizes any one of claims 1 to 12 the method when executing the computer program.

15. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program The step of method described in any one of claims 1 to 12 is realized when being executed by processor.