CN103699475A - Method, device and system for optimizing test samples in fuzzy test - Google Patents
Method, device and system for optimizing test samples in fuzzy test Download PDFInfo
- Publication number
- CN103699475A CN103699475A CN201210365720.9A CN201210365720A CN103699475A CN 103699475 A CN103699475 A CN 103699475A CN 201210365720 A CN201210365720 A CN 201210365720A CN 103699475 A CN103699475 A CN 103699475A
- Authority
- CN
- China
- Prior art keywords
- test case
- tested object
- test
- tested
- cases
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
- G06F21/577—Assessing vulnerabilities and evaluating computer system security
Abstract
The invention provides a method, a device and a system for ranking test samples. The method comprises the following steps that the following steps are executed for one turn or several turns; each test samples in a plurality of test samples is used for testing a test object according to the ranging sequence of the test samples in an initial test sample set, the attacking effect of each test sample is monitored, and in addition, the priority of each test sample is determined according to the attacking effect corresponding to each test sample; the test samples in the initial test sample set are re-ordered again according to the determined priority of each test sample; the re-ordered test sample set obtained in the current turn can be used as the initial test sample set used in the execution process in the next turn.
Description
Technical field
Present invention relates in general to software testing technology field, the test case relating more specifically to using in fuzz testing is optimized.
Background technology
The security test of software (comprising application program, communication software etc.) is actually a kind of detection in the software input space.The input space of software is by all possible input of this software or constituting of all possible input.Exhaustive is a kind of the most extreme method of verifying software behavior correctness, yet exhaustive do not possess operability concerning most of case, because generally, the input space of software program is very huge, may be even infinitely great.
Than exhaustive method, fuzz testing (Fuzz Testing), as a kind of random sample method, is a kind of more feasible Black-box Testing method.Fuzz testing carrys out work in the following manner: automatically for tested object (being application program, communication software etc.) provides random or half effectively input (being test case (Test Case)), to attempt triggering indicating fault and to make tested object collapse.This program mal or mistake can be indicated and be had Security Vulnerability.As a kind of Black-box Testing method, fuzz testing is generally used for large-scale software development project.At present, the most frequently used fuzz testing is based on character, and the input of random data stream (provide from file or other data stream, for example, provide from socket) as tested software program is provided the fuzz testing based on character.Fuzz testing based on character is just being widely used in the security test of agreement or network service.Fuzz testing is believed to improve the security of software, this be because: fuzz testing always can detect the not detectable leaks of some testers (bug), and these leaks tend to be ignored by software engineer.The main advantage of fuzz testing is: the cost of testing is relatively low, and can automatically realize completely; In addition, fuzz testing often can be found out leak more serious and that can be utilized, and these leaks can be used by victim.
Using a challenge of fuzz testing is how to utilize limited test resource to select effectively to detect the test case of leak.There are many methods to be used for selecting the test case for fuzz testing.A simple method is called Random-fuzzy test (RFT), and the method is selected test case randomly from input domain (that is, the set of all possible input).RFT has the following advantages: cost is low; Can automatically generate a large amount of test cases; And without software specifications and source code in the situation that generating test use case.In addition, RFT introduces test process by " randomness ".This randomness can reflect the confusion of system running environment best.Therefore, RFT can detect some leak that can not expose by deterministic method.All these advantages are widely used RFT in the industry of finding software vulnerability.
Yet, in RFT, do not attempt with any can with information guide test.Because present software at least has the validity that basic mechanism checks input, so too special " fuzzy " data significantly different from legal input will easily be detected and be dropped.In addition, in RFT, may there is very much the test case of repetition, because some test cases have identical attack effect for target software.Therefore, the major defect of RFT is that efficiency is poor.
Summary of the invention
According to an aspect of the present invention, a kind of method for test case is sorted is provided, comprise: carry out following steps one and take turns or take turns more: according to putting in order of a plurality of test cases in initial test case set, use each test case in described a plurality of test case to test tested object, monitor the attack effect of each test case, and according to determine the priority of each test case corresponding to the attack effect of each test case; According to the priority of determined each test case, to a plurality of test case rearrangements in described test case set; The set of the test case after the rearrangement wherein, obtaining in current round can be as the initial test case set of using in the execution of next one.
Further, in described method, described one or more standard at least comprises one of the following: test case can cause tested object to break down individually, test case can jointly cause tested object to break down with other test cases, test case reaches predetermined extent with the similar degree of the test case that can cause individually tested object to break down, and test case causes tested object operating lag.
Further, in described method, by priority value, represent priority, when the attack effect of test case meets described one or more standard, in the current priority value of test case, increase corresponding priority value.
Further, in described method, increase corresponding priority value and comprise: when a test case can cause tested object to break down individually, for this test case increases by the first high priority value; When a test case can jointly cause tested object to break down with other test cases, for this test case increases by the second high priority value; When a test case reaches predetermined extent with the similar degree that can cause individually the test case that tested object breaks down, it is the priority value that this test case increases third high; When a test case causes tested object operating lag, for this test case increases by the 4th high priority value.
Further, in described method, according to putting in order of a plurality of test cases in initial test case set, use each test case in described a plurality of test case that tested object is tested and is specially: the continuous application test case that puts in order according to a plurality of test cases in initial test case set is tested tested object, until described tested object breaks down after a test case of application, described startup tested object use test case described in last of application to retest this tested object again, if described tested object breaks down again, last test case that judges described application can cause tested object to break down individually, otherwise last test case that judges described application can jointly cause tested object to break down with other test cases.
Further, in described method, this test case of described judgement can jointly cause tested object to break down with other test cases comprising: (a) restart tested object; (b) use this test case j test case and this test case before to retest this tested object; Above step (a) and (b) is carried out in circulation, until tested object breaks down, now judge that j test case before this test case and this test case can jointly cause tested object to break down, wherein, the initial value of j is 1, and after each circulation, the value of j adds 1.
Further, in described method, after tested object application testing use-case, if do not receive the response of tested object, to tested object, send normal input, if still do not receive response, judge that tested object breaks down; If receive correspondingly, according to the response time, judge whether current test case causes operating lag.
Further, in described method, described tested object is communication software.
According to another aspect of the present invention, a kind of device for test case is sorted is provided, comprise: fault monitoring module, for monitoring putting in order according to a plurality of test cases of initial test case set, use the attack effect of the test that each test case in described a plurality of test case carries out tested object, and according to the attack effect corresponding to each test case, determine the priority of each test case; Optimize module, for according to the priority of determined each test case, described a plurality of test cases are resequenced.
Further, in described device, described one or more standard at least comprises one of the following: test case can cause tested object to break down individually, test case can jointly cause tested object to break down with other test cases, test case reaches predetermined extent with the similar degree of the test case that can cause individually tested object to break down, and test case causes tested object operating lag.
Further, in described device, by priority value, represent priority, when the attack effect of test case meets a standard in described one or more standard, in the current priority value of test case, increase corresponding priority value.
Further, in described device, increase corresponding priority value and comprise: when a test case can cause tested object to break down individually, for this test case increases by the first high priority value; When a test case can jointly cause tested object to break down with other test cases, for this test case increases by the second high priority value; When a test case reaches predetermined extent with the similar degree that can cause individually the test case that tested object breaks down, it is the priority value that this test case increases third high; When a test case causes tested object operating lag, for this test case increases by the 4th high priority value.
Further, in described device, described fault monitoring module comprises: autoboot module, for when tested object breaks down, restarts tested object; Wherein, when application one test case, tested object breaks down, by described autoboot module, restarting tested object and using this test case to retest after this tested object, if this tested object breaks down again, described fault monitoring module judges that this test case can cause tested object to break down individually, otherwise judges that this test case can jointly cause tested object to break down with other test cases.
Further, in described device, this test case of described judgement can jointly cause tested object to break down with other test cases comprising: (a) restart tested object; (b) use this test case j test case and this test case before to retest this tested object; Above step (a) and (b) is carried out in circulation, until tested object breaks down, now judge that j test case before this test case and this test case can jointly cause tested object to break down, wherein, the initial value of j is 1, and after each circulation, the value of j adds 1.
Further, in described device, described fault monitoring module comprises viability detecting module, be used for after tested object application testing use-case, if do not receive the response of tested object, to tested object, send normal input, if still do not receive response, judge that tested object breaks down; If receive response, according to the response time, judge whether current test case causes operating lag.
Further, in described device, described tested object is communication software.
According to a further aspect of the invention, provide a kind of method for fuzz testing, having comprised: generate a plurality of test cases; Carrying out following steps one takes turns or takes turns more: monitor putting in order according to a plurality of test cases in initial test case set, use the attack effect of the test that each test case in described a plurality of test case carries out tested object, and according to the attack effect corresponding to each test case, determine the priority of each test case; According to the priority of determined each test case, to described a plurality of test case rearrangements; The set of the test case after the rearrangement wherein, obtaining in current round can be as the initial test case set of using in the execution of next one.
According to another aspect of the present invention, provide a kind of system for fuzz testing, having comprised: test case generator, for generating randomly a plurality of test cases; Test case storer, for described a plurality of test cases that storage generates in order; Fuzz testing device, for according to the storage order of described a plurality of test cases, tests tested object by each test case in described a plurality of test cases; Fault monitor, for monitor test process, and according to the attack effect corresponding to each test case, determines the priority of each test case; Optimizer, for according to the priority of each test case of determined described a plurality of test cases, to described a plurality of test case rearrangements, and stores back described test case storer by the described a plurality of test cases after rearrangement.
The method that the application of the invention provides, can automatically optimize the test case of random generation, so that can greatly improve the Hole Detection efficiency of RFT.The optimizing process of the application's optimization method is adaptive, and can optimize test case according to attack effect.
These and other characteristic of the present invention, Characteristics and advantages will become obviously in the instructions being described with reference to the drawings, and wherein accompanying drawing illustrates principle of the present invention in the mode of example.Instructions is only for illustrative purposes, and does not limit the scope of the invention.The reference diagram below quoted all refers to accompanying drawing.
Accompanying drawing explanation
By reference to the explanation providing below in conjunction with accompanying drawing, it is more apparent that various aspects described herein will become, wherein:
Fig. 1 is the process flow diagram of method according to an embodiment of the invention;
Fig. 2 is the schematic diagram of device according to an embodiment of the invention;
Fig. 3 is the process flow diagram of method according to an embodiment of the invention;
Fig. 4 is process flow diagram of recalling the method for test according to an embodiment of the invention;
Fig. 5 is the schematic diagram of system according to an embodiment of the invention.
Should be understood that, in all above-mentioned accompanying drawings, same Reference numeral refers to same, similar or characteristic of correspondence or function.
Embodiment
The application will be described for specific embodiment and with reference to certain figures, but the application is not limited to this, and the application limits by claim.Described accompanying drawing is only illustrative and not restrictive.In the accompanying drawings, for ease of explanation, the size of some elements may be exaggerated and not drawn on scale.For the indefinite article using when mentioning singular noun or definite article, for example " one ", " one " and " should/described ", except as otherwise outside clear and definite regulation, this comprises the plural form of this noun.
In the former research that Random-fuzzy is tested, the most often with two kinds of yardsticks, assess the validity of test case set, a kind of yardstick is the possibility (being called PF yardstick) that at least one leak detected, and another kind is the quantity (being called EF yardstick) of estimating the leak detect.Although these two kinds of yardsticks are very popular, they still have many deficiencies.For example, for EF yardstick, higher EF yardstick must not mean can detect more inefficacy or how different leaks; For PF yardstick, it does not have reflection the difference between the use-case of leak of varying number can be detected.Therefore, the yardstick of above-mentioned two kinds of assessment validity is in many cases and not bery desirable.
In the present invention, with the different yardstick of a kind of and aforementioned two kinds of validity yardsticks, assess validity, that is, use " the expectation quantity of the required test case of first leak being detected " this validity yardstick (being called FF yardstick).FF yardstick can more naturally, more directly reflect the validity of Test Strategy.If the value of FF yardstick is lower, mean that corresponding Test Strategy is more effective, this is because only need test case still less just can expose first leak.In practice, when leak being detected, conventionally can stop testing and starting debugging, only have after repairing described leak and just restart test phase.Therefore,, in actual fuzz testing, the test case set that first leak can be detected is faster more effective often.
The invention provides a kind of adaptive approach, for Random-fuzzy test, the method can realize less FF.General plotting of the present invention is: the attack effect based on random test use-case sorts to these test cases, thereby makes can by the test case of most probable exposure software vulnerability, test as early as possible in test subsequently.
Fig. 1 shows the process flow diagram of the method that test case is sorted 100 according to an embodiment of the invention.A set of the given a plurality of test cases that comprise random generation, and as the software program of tested object, for example tested object can, for a communication protocol, sort to test case by the method shown in Fig. 1.
In step S101, according to the putting in order of test case (putting in order of test case is putting in order while generating at random for the first time time), by all test cases, software program to be measured is tested, detect the attack effect of each test case, and according to the attack effect corresponding to each test case for set in each test case determine corresponding priority.Can assess attack effect by one or more standards, for example, whether a test case can cause break down (for example, to the response of request nothing) of tested software individually, and whether a test case can cause breaking down of tested software jointly with other test cases.In addition, test case can be divided into two or more priority.
In step S102, according to the priority of determined each test case, to test case rearrangement, thus the set of the test case after being resequenced.When the order according to after rearrangement, while testing software program of the same type by the test case of described generation, can find quickly first place's leak.For example, when the test case set after using sequence is tested operating in the software program of same software program on another machine or another version, can find quickly first place's leak.。
In above embodiment, the step of test case being carried out taking turns sequence has only been described.In another embodiment, can carry out to test case the sequence of two-wheeled or more wheels, and take turns and in sequence, carry out identical step at each.That is to say, can cyclically perform step S101 and S102, in circulation each time, set using the set of the test case after the rearrangement obtaining in last round as the initial test case of using in the sequence of next one, another tested software program is tested, and according to test result minor sort again.Conventionally, if the round of circulation is more, more accurate to the optimization of the set of the test case of initial generation, make more may cause the test case that tested software breaks down to come more forward position.When software program is tested by the test case of arranging in order obtaining thus, can be by the test case of smaller amounts, disclose leak quickly.
Fig. 2 shows the device that test case is sorted 200 according to an embodiment of the invention, and it can realize method as shown in Figure 1.As shown in the figure, device 200 comprises fault monitoring module 201 and optimizes module 202.Fault monitoring module 201 is for monitoring putting in order according to a plurality of test cases of initial test case set, the attack effect that uses each test case in described a plurality of test case to test tested object, and according to the attack effect corresponding to each test case, determine the priority of each test case.Optimize module 202 for according to the priority of each test case of determined described a plurality of test cases, described a plurality of test cases are resequenced.
Fig. 3 shows the method that test case is sorted 300 according to an embodiment of the invention.In Fig. 3, show the set T={t that uses m random test use-case
1, t
2..., t
mcarry out R
jthe idiographic flow of (j>=1) wheel test.In test process, by according to the attack effect of test case, be each test case t
idetermine corresponding priority.Can represent priority with priority value, and can adopt a plurality of standards to weigh the attack effect of test case.In each of this embodiment, taking turns in test, can be that each test case is distributed the priority value corresponding to four kinds of standards as described below:
(1) cause individually breaking down: the test case that tested software breaks down that causes that can be independent is assigned with highest priority value PRI_IDV;
(2) jointly cause breaking down: software fault is caused jointly by a plurality of test cases sometimes, for these test cases are distributed the second corresponding high priority value PRI_CLB;
(3) similar with the test case that causes individually breaking down: to have with the test case of the similar content of test case that can cause individually breaking down and can regard be assigned with corresponding third high priority value PRI_IDV_SIM(as all test cases as character string, therefore can adopt for example sequence alignment algorithm to measure two similarities between test case, if similarity reaches predetermined threshold value, test case can be marked as this priority so);
(4) operating lag: cause the test case of the response of delay to be considered to likely trigger fault, for this test case distributes the 4th corresponding high priority value PRI_RSP(when tested program is communication protocol software, this standard is particularly useful, for example, if the response time of software program has surpassed the average response time of last round of test, can think operating lag has occurred so).
In one takes turns test, be that the priority value that test case is distributed can be accumulated on the current priority value having of this test case, the initial priority value of each test case can be set to identical numerical value, and for example 0.Can adopt different numerical value to represent different priority, in the embodiments of figure 3, for each test case is distributed corresponding priority value p
i, p
ivalue PRI_IDV, PRI_CLB, PRI_IDV_SIM and PRI_RSP can be respectively 4,3,2,1, can be also 8,6,4,2 etc.
As shown in Figure 3, each is taken turns test and starts from step S301, and for example current test is R
jwheel.At step S301, from wherein having by p
iin the test case set T of a plurality of test cases of value size sequence, select in order a test case t
ibe applied to tested software program, then perform step S302.
Whether the tested software program of having applied test case in step S302 judgement is also in active state, and for example whether for example,, by sending normal input (file of standard or packet) to tested software program, observing tested software program has response.If response, does not show that tested software program breaks down and continues to perform step S305; If there is response, show that tested software program does not break down and continues to perform step S303.Normal input is that the method that obtains normal input is a lot of as long as tested software is being survived and just will inevitably made the input of response, and a kind of method is to use software (except tested software) and the tested software communication of legal a, standard.During this, their communication is normal (because protocol compliant standard) certainly.The instrument such as use amount Wireshark or Tcpdump can capture normal input.Can also construct by hand the normal input that meets tested software protocol specification in addition.
At step S305, judge whether the fault of tested software program is caused by independent test case.If, by current test case t
ipriority value p
ibe updated to p
i=p
i+ PRI_IDV, then performs step S310; If not, by current test case t
ipriority value p
ibe updated to p
i=p
i+ PRI_CLB, then performs step S310.For example, can come in the following way failure judgement whether by independent test case, to be caused: the continuous application test case that puts in order according to a plurality of test cases in initial test case set is tested tested object, until described tested object breaks down after a test case of application.Now, again described startup tested object use test case described in last of application to retest this tested object, if described tested object breaks down again, last test case that judges described application can cause tested object to break down individually, otherwise judges that last test case of described application can jointly cause tested object to break down with other test cases.
At step S303, judge that whether current test case is similar with the test case that can trigger individually fault.If similar, perform step S308, otherwise execution step S304.
At step S308, by current test case t
ipriority value p
ibe updated to p
i=p
i+ PRI_SIM_CLB, then performs step S310.
At step S304, judge whether the response time surpasses threshold value RT
j-1.For example, value RT
j-1it can be the average response time of the test case calculated in last round of test.If surpassed threshold value RT
j-1, at step S309 by current test case t
ipriority p
ibe updated to p
i=p
i+ PRI_RSP, then performs step S310.If do not surpass threshold value, directly perform step S310.
At step S310, judge whether current test case is last test case in set T, application in epicycle test.If not last test case, turn back to step S301, to select next test case to test; If last test case performs step S311.
At step S311, calculate in epicycle test and gather all test case t in T
iaverage response time RT
j, then perform step S312.When actual computation, for example, can only according to the response time that has produced the test case of response, calculate RT
j.
At step S312, according to each test case t
icurrent priority value p
i, the test case rearrangement by set T, then finishes R
jthe test of wheel.
More than described the test case set T application one detailed method flow process of taking turns test, but in other embodiments, can be to the many wheel tests of test case set T application.When carrying out R
jduring the test of wheel, will be according at R
j-1test case in the order application T obtaining after the test case in T being resequenced after wheel test.
In an embodiment of method shown in execution graph 3, start tested object, then to operating tested object application testing use-case, if tested object does not break down after a test case of application, the next test case of application can be in the situation that keeping tested object operation, continued, and this tested object need not be restarted.If tested object breaks down after a test case of application, restart this tested object, and use this test case to retest this tested object, if this tested object breaks down again, judge that this test case can cause tested object to break down individually, otherwise judge that this test case can jointly cause tested object to break down with other test cases of previously having used.Also a plurality of test cases be can recall and the test case that jointly causes tested object to break down, the test process of recalling a plurality of test cases with an example explanation below in conjunction with Fig. 4 determined.
For example, with the communication software of i testing case in a test case set, if communication software has response, apply i+1 this communication software of test case continuation operation and test.If software is without response after applying i test case, to this software, send normal input, if software is to the normal input nothing response sending, determine that software produces fault.
In the case, restart this tested communication software, then apply separately i testing case software.Now, if communication software, without response, sends normal input to this software, if software is to the normal input nothing response sending, determine that software produces fault, and reach a conclusion and by i test case, triggered individually when prior fault, so increase by the first high priority value for it.Otherwise if while applying i test case separately, communication software has response to i test case, restart software continuous application i-1 and i test case.If software is without response, send normal input, if software is to the normal input nothing response sending, determine that software produces fault, and reach a conclusion when prior fault is by i-1 and i common triggering of test case, therefore for the two all increases by the second high priority value.
If software has response to i-1 with i test case, restart software, and continuous application i-2 is individual, i-1 is individual and i test case.If software is without response, send normal input, if software to normal input also without response, determine that software produces fault, and determine when prior fault is by i-2, i-1 and i common triggering of test case, thereby give i-2, i-1 is individual and i the priority value that test case second is high.
Otherwise, if software is individual to i-2, i-1 is individual and i test case has response, restart software, since i-3 test case again application testing use-case, repeat above-mentioned steps, until find all test cases of common triggering fault, and give them the second high priority value.But, oversize for preventing test loop, take into account testing efficiency, can set the upper limit number of this circulation.If for example the upper limit number of circulation is set as to 10,, in above-mentioned example, only date back to i-10 test case.
In the embodiment describing in conjunction with Fig. 3, adopt four standards as above to assess the attack effect of described test case, thereby distribute corresponding priority value.But evaluation criteria is not limited to above Four types.In other embodiments, can adopt the standard of other quantity or other types to assess attack effect, thereby the quantity of the priority of dividing also can be different.
In the embodiment describing in conjunction with Fig. 3, by send normal input to tested software program, determine whether tested software program fault has occurred veritably.But, in other embodiments, can adopt other modes to judge whether measurand breaks down.
In the embodiment describing in conjunction with Fig. 3, utilize the threshold value of the average response time of the test case of calculating in last round of test as the response time.But, in other embodiments, can adopt other values as threshold value.
Below with set T={t
1, t
2, t
3, t
4, t
5, t
6be example, after the method shown in application Fig. 3, institute can getable result.T
1to t
6order be the initial orders that generate of these test cases, before using them to test, their priority value p separately
ibe all 0.While testing in the first round, according to their initial orders generating, these test cases are applied to tested software program one by one.Taking turns after test test case t through one
1to t
6priority value become respectively p
1=0, p
2=0, p
3=3, p
4=4, p
5=2, p
6=1.That is to say, when using t
1and t
2, while testing first tested software program, they all cannot cause any one situation in four kinds of standards as above to occur; When using t
3while testing first tested software program, program breaks down, and this fault may be relevant with the test case of operation before; When using t
4while testing first tested software program, can cause individually program to break down; And test case t
5with t
4very similar; When using t
6while testing first tested software program, cause tested program operating lag.Therefore,, according to above-mentioned priority value, the test case in test case set T will be reordered as { t
4, t
3, t
5, t
6, t
1, t
2.When other tested objects are tested by the test case after rearrangement, will likely find quickly first leak, thereby improve testing efficiency.
After above-mentioned test set T being carried out to first round test sequence, can also be on its basis, set of applications T carries out second and takes turns test again.Second, take turns in test, according to t
4, t
3, t
5, t
6, t
1, t
2order for example, to second tested software program (program obtaining by revising first tested software program, or the program different from first tested software program version etc.) application testing use-case.Second, take turns in test, distribute to test case t
4, t
3, t
5, t
6, t
1, t
2priority value will be respectively 4,0,2,4,0,1.That is to say, second, take turns in test, when using t
4and t
6during second tested software program of test, all can cause individually program to break down; And test case t
5with t
4very similar; When using t
3and t
1during second tested software program of test, all cannot cause any one situation in four kinds of standards as above to occur; When using t
2during second tested software program of test, cause tested program operating lag.Thereby, through after two-wheeled test, corresponding to total priority value of each test case, will become p
1=0, p
2=1, p
3=3, p
4=8, p
5=4, p
6=5.Therefore,, after two-wheeled test, the order of the priority of each test case is by the big or small order of the total priority value corresponding to as above.Therefore, now, the order after the test case in test case set T is according to priority resequenced will be { t
4, t
6, t
5, t
3, t
2, t
1.When other tested objects are tested by the test case after rearrangement, may find quickly first leak than the test case after the sequence of the use first round, thereby improve testing efficiency.
More than describe two-wheeled test or sequence, in practice, can carry out the sequence of more wheels to test case set T.
Fig. 5 shows the schematic diagram of the system 500 for fuzz testing according to an embodiment of the invention.As shown in Figure 5, system 500 comprises test case generator 510, and for generating randomly a plurality of test cases, the plurality of test case forms test case set T, the test case set T for example describing in above-mentioned each embodiment.
System 500 also comprises test case storer 520, for storing in order a plurality of test cases of described generation.
System 500 also comprises fuzz testing device 530, be used for according to the storage order of a plurality of test cases of test case storer 520, by each test case in described a plurality of test cases, tested object is tested one by one, for example, operation tested software program, and input above-mentioned test case to tested software program.
System 500 also comprises fault monitor 540, for monitoring fuzz testing process, and according to the attack effect corresponding to each test case, determines the corresponding priority of each test case.Described fault monitor 540 may further include viability detecting module 541 and autoboot module 542.Wherein, autoboot module 542 for example,, for when tested object breaks down (, tested software program is without any when response), restarts tested object.Viability detecting module 541 is for for example, described test period (after tested object has been inputted test case), to tested object, send normal input, and if receive response, according to the response time, judge whether current test case causes operating lag; If do not receive response, judge that current test case causes tested object to break down.When application one test case, tested object breaks down, can restart tested object by autoboot module 542, and use this test case to retest this tested object, if this tested object breaks down again, judge that this test case can cause tested object to break down individually, otherwise judge that this test case can jointly cause tested object to break down with other test cases.
In addition, system 500 also comprises optimizer 550, for basis, distribute to the priority of each test case of described a plurality of test cases, to described a plurality of test case rearrangements, and the described a plurality of test cases after rearrangement are stored back to described test case storer 520, thereby complete one, take turns the renewal that test case is put in order.By the tactic test case after upgrading, tested object is tested next time, and can be carried out the renewal of more wheels to the test case of storage in storer 520.
The present invention has below been described in conjunction with specific embodiments.Those skilled in the art it will also be appreciated that program or design code (can be referred to as for convenience's sake " software " or " software module " herein) or both combinations of the various forms of include instructions that can be embodied as electronic hardware in connection with various illustrative logical blocks, module, unit, device, circuit and the algorithm steps of each side description disclosed herein, be carried out by processor.For this interchangeability of hardware and software is clearly described, generally from their functional angles, various example components, frame, module, unit and step are described above.These functions are embodied as to hardware, firmware or software to be depended on specific application and is applied to the design constraint in whole system.Technician can realize described function in a different manner for every kind of application-specific, but this realization should not determined to be interpreted as causing departing from the scope of the present disclosure.
It should be noted, providing above-described embodiment is unrestricted the present invention in order to describe the present invention, the special characteristic comprising at above-mentioned each embodiment is not limited to only comprise in this particular example, for the feature in different embodiment, can carry out appropriately combinedly, and not depart from the scope of the present invention.And it being understood that those skilled in the art are easy to expect can take modifications and changes without departing from the spirit and scope of the present invention.This modifications and changes are regarded as in the scope of the present invention and claims.Protection scope of the present invention is defined by the claims.In addition, any Reference numeral in claims all should not be interpreted as limitations on claims.
Claims (18)
1. the method for test case is sorted, comprising:
Carrying out following steps one takes turns or takes turns more:
According to putting in order of a plurality of test cases in initial test case set, use each test case in described a plurality of test case to test tested object, monitor the attack effect of each test case, and according to determine the priority of each test case corresponding to the attack effect of each test case;
According to the priority of determined each test case, to a plurality of test case rearrangements in described test case set;
The set of the test case after the rearrangement wherein, obtaining in current round can be as the initial test case set of using in the execution of next one.
2. the method for claim 1, wherein, adopt one or more standards to assess described attack effect, described one or more standard at least comprises one of the following: test case can cause tested object to break down individually, test case can jointly cause tested object to break down with other test cases, test case reaches predetermined extent with the similar degree of the test case that can cause individually tested object to break down, and test case causes tested object operating lag.
3. method as claimed in claim 2, wherein, represents priority by priority value, when the attack effect of test case meets described one or more standard, in the current priority value of test case, increases corresponding priority value.
4. method as claimed in claim 3 wherein, increases corresponding priority value and comprises in the current priority value of test case:
When a test case can cause tested object to break down individually, for this test case increases by the first high priority value;
When a test case can jointly cause tested object to break down with other test cases, for this test case increases by the second high priority value;
When a test case reaches predetermined extent with the similar degree that can cause individually the test case that tested object breaks down, it is the priority value that this test case increases third high;
When a test case causes tested object operating lag, for this test case increases by the 4th high priority value.
5. the method as described in any one in claim 2-4, wherein, according to putting in order of a plurality of test cases in initial test case set, use each test case in described a plurality of test case that tested object is tested and is specially: the continuous application test case that puts in order according to a plurality of test cases in initial test case set is tested tested object, until described tested object breaks down after a test case of application, described startup tested object use test case described in last of application to retest this tested object again, if described tested object breaks down again, last test case that judges described application can cause tested object to break down individually, otherwise last test case that judges described application can jointly cause tested object to break down with other test cases.
6. method as claimed in claim 5, wherein, last test case of described judgement application can jointly cause tested object to break down with other test cases further comprising:
(a) restart tested object;
(b) use last test case and tested object described in its j testing case before of described application;
Above step (a) and (b) is carried out in circulation, until tested object breaks down, last test case that now judges described application with and j test case before can jointly cause tested object to break down, wherein, the initial value of j is 1, and after each circulation, the value of j adds 1.
7. method as claimed in claim 2, wherein, after tested object application testing use-case, if do not receive the response of tested object, sends normal input to tested object, if still do not receive response, judges that tested object breaks down; If receive response, according to the response time, judge whether current test case causes operating lag.
8. the method for claim 1, wherein described tested object is communication software.
9. the device for test case is sorted, comprising:
Fault monitoring module, for monitoring putting in order according to a plurality of test cases of initial test case set, use the attack effect that in described a plurality of test case, each test case is tested tested object, and according to the attack effect corresponding to each test case, determine the priority of each test case;
Optimize module, for according to the priority of determined each test case, a plurality of test cases in described test case set are resequenced.
10. device as claimed in claim 9, wherein, adopt one or more standards to assess described attack effect, described one or more standard at least comprises one of the following: test case can cause tested object to break down individually, test case can jointly cause tested object to break down with other test cases, test case reaches predetermined extent with the similar degree of the test case that can cause individually tested object to break down, and test case causes tested object operating lag.
11. devices as described in claim 9 or 10, wherein, represent priority by priority value, when the attack effect of test case meets described one or more standard, in the current priority value of test case, increase corresponding priority value.
12. devices as claimed in claim 11, wherein, increase corresponding priority value and comprise:
When a test case can cause tested object to break down individually, for this test case increases by the first high priority value;
When a test case can jointly cause tested object to break down with other test cases, for this test case increases by the second high priority value;
When a test case reaches predetermined extent with the similar degree that can cause individually the test case that tested object breaks down, it is the priority value that this test case increases third high;
When a test case causes tested object operating lag, for this test case increases by the 4th high priority value.
13. devices as claimed in claim 9, described fault monitoring module comprises autoboot module, it,, for when tested object breaks down, restarts tested object, described fault monitoring module is used each test case in described a plurality of test cases that tested object is tested and is specially according to putting in order of a plurality of test cases in initial test case set, the continuous application test case that puts in order according to a plurality of test cases in initial test case set is tested tested object, until described tested object breaks down after a test case of application, utilize autoboot module described startup tested object again, and use test case described in last of application to retest this tested object, if described tested object breaks down again, last test case that judges described application can cause tested object to break down individually, otherwise last test case that judges described application can jointly cause tested object to break down with other test cases.
14. devices as claimed in claim 13, wherein, this test case of described judgement can jointly cause tested object to break down with other test cases comprising:
(a) restart tested object; ,
(b) use this test case and j this tested object of testing case before it;
Above step (a) and (b) is carried out in circulation, until tested object breaks down, now judge that j test case before this test case and this test case can jointly cause tested object to break down, wherein, the initial value of j is 1, and after each circulation, the value of j adds 1.
15. devices as claimed in claim 9, described fault monitoring module comprises:
Viability detecting module, for after tested object application testing use-case, if do not receive the response of tested object, sends normal input to tested object, if still do not receive response, judges that tested object breaks down; If receive response, according to the response time, judge whether current test case causes operating lag.
16. devices as claimed in claim 9, wherein, described tested object is communication software.
17. 1 kinds of methods for fuzz testing, comprising:
Generate a plurality of test cases;
Carrying out following steps one takes turns or takes turns more:
According to putting in order of a plurality of test cases in initial test case set, use each test case in described a plurality of test case to test tested object, monitor the attack effect of each test case, and according to determine the priority of each test case corresponding to the attack effect of each test case;
According to the priority of determined each test case, to a plurality of test case rearrangements in described test case set;
The set of the test case after the rearrangement wherein, obtaining in current round can be as the initial test case set of using in the execution of next one.
18. 1 kinds of systems for fuzz testing, comprising:
Test case generator, for generating randomly a plurality of test cases;
Test case storer, for described a plurality of test cases that storage generates in order;
Fuzz testing device, for according to the storage order of described a plurality of test cases, tests tested object by each test case in described a plurality of test cases;
Fault monitor, for monitor test process, and according to determine the priority of each test case corresponding to the attack effect of each test case;
Optimizer, for according to the priority of each test case of determined described a plurality of test cases, to described a plurality of test case rearrangements, and stores back described test case storer by the described a plurality of test cases after rearrangement.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210365720.9A CN103699475B (en) | 2012-09-27 | 2012-09-27 | The method that test case in fuzz testing is optimized, device and system |
| PCT/EP2013/070164 WO2014049104A1 (en) | 2012-09-27 | 2013-09-27 | A method, apparatus and system for optimizing test cases in fuzz testing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210365720.9A CN103699475B (en) | 2012-09-27 | 2012-09-27 | The method that test case in fuzz testing is optimized, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103699475A true CN103699475A (en) | 2014-04-02 |
| CN103699475B CN103699475B (en) | 2016-09-28 |
Family
ID=49301465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210365720.9A Active CN103699475B (en) | 2012-09-27 | 2012-09-27 | The method that test case in fuzz testing is optimized, device and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN103699475B (en) |
| WO (1) | WO2014049104A1 (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105487966A (en) * | 2014-09-17 | 2016-04-13 | 腾讯科技（深圳）有限公司 | Program testing method, device and system |
| CN108388509A (en) * | 2018-02-07 | 2018-08-10 | 平安科技（深圳）有限公司 | A kind of method for testing software, computer readable storage medium and terminal device |
| CN108449234A (en) * | 2018-04-04 | 2018-08-24 | 国家计算机网络与信息安全管理中心 | A kind of intelligent fuzzy test method based on dynamic analysis |
| CN108490922A (en) * | 2018-04-27 | 2018-09-04 | 北京新能源汽车股份有限公司 | A kind of generation method and device of unified diagnostic service test case |
| CN108959079A (en) * | 2018-06-27 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of with automatic test is leading software agile development method and system |
| CN109120643A (en) * | 2018-10-11 | 2019-01-01 | 北京知道创宇信息技术有限公司 | Penetration test method and device |
| CN109726124A (en) * | 2018-12-20 | 2019-05-07 | 北京爱奇艺科技有限公司 | Test macro, test method, managing device, test device and calculating equipment |
| CN110716869A (en) * | 2019-09-18 | 2020-01-21 | 平安科技（深圳）有限公司 | Test case generation method and device, electronic equipment and computer readable storage medium |
| CN113157551A (en) * | 2021-01-14 | 2021-07-23 | 首都师范大学 | ROS-oriented differential fuzzy test method |
| CN113419954A (en) * | 2021-06-25 | 2021-09-21 | 陕西浪潮英信科技有限公司 | Execution method and system of automation use case of cloud operating system and related components |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104375942B (en) * | 2014-12-11 | 2017-02-08 | 无锡江南计算技术研究所 | Binary oriented hybrid fuzzing method |
| US10176426B2 (en) | 2015-07-07 | 2019-01-08 | International Business Machines Corporation | Predictive model scoring to optimize test case order in real time |
| CN107644164B (en) * | 2016-07-21 | 2020-05-12 | 中国电信股份有限公司 | Vulnerability mining method and device |
| US10831646B2 (en) | 2019-01-02 | 2020-11-10 | International Business Machines Corporation | Resources usage for fuzz testing applications |
| US10380350B1 (en) * | 2019-01-15 | 2019-08-13 | Cyberark Software Ltd. | Efficient and comprehensive source code fuzzing |
| CN110995770B (en) * | 2020-03-02 | 2020-06-16 | 信联科技(南京)有限公司 | Fuzzy test application effect comparison method |
| CN111611156B (en) * | 2020-04-28 | 2024-01-30 | 北京小米移动软件有限公司 | Function test method, function test device, and computer-readable storage medium |
| CN112035343B (en) * | 2020-08-13 | 2022-02-01 | 武汉大学 | Test case generation method and system based on Bayesian estimation |
| CN114978974B (en) * | 2022-05-20 | 2023-09-19 | 国网重庆市电力公司电力科学研究院 | Wireless communication module testing method, device, equipment and storage medium |
| CN116594884B (en) * | 2023-05-16 | 2023-10-27 | 上海安般信息科技有限公司 | Fuzzy test method and device based on concurrent mode |
| CN116881171B (en) * | 2023-09-05 | 2023-12-22 | 中科方德软件有限公司 | Seed use case processing method, device, equipment and storage medium in fuzzy test |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090265681A1 (en) * | 2008-04-21 | 2009-10-22 | Microsoft Corporation | Ranking and optimizing automated test scripts |
| WO2010069587A1 (en) * | 2008-12-18 | 2010-06-24 | Nec Europe Ltd. | Method and device for supporting penetration testing of a computer system |
| CN101859273A (en) * | 2009-04-07 | 2010-10-13 | 西门子（中国）有限公司 | Method and device for generating test cases |
-
2012
- 2012-09-27 CN CN201210365720.9A patent/CN103699475B/en active Active
-
2013
- 2013-09-27 WO PCT/EP2013/070164 patent/WO2014049104A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090265681A1 (en) * | 2008-04-21 | 2009-10-22 | Microsoft Corporation | Ranking and optimizing automated test scripts |
| WO2010069587A1 (en) * | 2008-12-18 | 2010-06-24 | Nec Europe Ltd. | Method and device for supporting penetration testing of a computer system |
| CN101859273A (en) * | 2009-04-07 | 2010-10-13 | 西门子（中国）有限公司 | Method and device for generating test cases |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105487966A (en) * | 2014-09-17 | 2016-04-13 | 腾讯科技（深圳）有限公司 | Program testing method, device and system |
| CN108388509B (en) * | 2018-02-07 | 2020-07-03 | 平安科技（深圳）有限公司 | Software testing method, computer readable storage medium and terminal equipment |
| CN108388509A (en) * | 2018-02-07 | 2018-08-10 | 平安科技（深圳）有限公司 | A kind of method for testing software, computer readable storage medium and terminal device |
| CN108449234A (en) * | 2018-04-04 | 2018-08-24 | 国家计算机网络与信息安全管理中心 | A kind of intelligent fuzzy test method based on dynamic analysis |
| CN108490922A (en) * | 2018-04-27 | 2018-09-04 | 北京新能源汽车股份有限公司 | A kind of generation method and device of unified diagnostic service test case |
| CN108490922B (en) * | 2018-04-27 | 2020-03-24 | 北京新能源汽车股份有限公司 | Generation method and device of unified diagnostic service test case |
| CN108959079A (en) * | 2018-06-27 | 2018-12-07 | 郑州云海信息技术有限公司 | A kind of with automatic test is leading software agile development method and system |
| CN108959079B (en) * | 2018-06-27 | 2021-08-20 | 郑州云海信息技术有限公司 | Software agile development method and system taking automatic test as leading factor |
| CN109120643A (en) * | 2018-10-11 | 2019-01-01 | 北京知道创宇信息技术有限公司 | Penetration test method and device |
| CN109120643B (en) * | 2018-10-11 | 2020-11-20 | 北京知道创宇信息技术股份有限公司 | Penetration testing method and device |
| CN109726124A (en) * | 2018-12-20 | 2019-05-07 | 北京爱奇艺科技有限公司 | Test macro, test method, managing device, test device and calculating equipment |
| CN110716869A (en) * | 2019-09-18 | 2020-01-21 | 平安科技（深圳）有限公司 | Test case generation method and device, electronic equipment and computer readable storage medium |
| CN113157551A (en) * | 2021-01-14 | 2021-07-23 | 首都师范大学 | ROS-oriented differential fuzzy test method |
| CN113157551B (en) * | 2021-01-14 | 2022-09-09 | 首都师范大学 | ROS-oriented differential fuzzy test method |
| CN113419954A (en) * | 2021-06-25 | 2021-09-21 | 陕西浪潮英信科技有限公司 | Execution method and system of automation use case of cloud operating system and related components |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2014049104A1 (en) | 2014-04-03 |
| CN103699475B (en) | 2016-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103699475A (en) | Method, device and system for optimizing test samples in fuzzy test | |
| US9672085B2 (en) | Adaptive fault diagnosis | |
| US9451017B2 (en) | Method and system for combining trace data describing multiple individual transaction executions with transaction processing infrastructure monitoring data | |
| CN107807877B (en) | Code performance testing method and device | |
| CN110618924B (en) | Link pressure testing method of web application system | |
| CN110888783A (en) | Monitoring method and device of micro-service system and electronic equipment | |
| CN108763089B (en) | Test method, device and system | |
| TW201636839A (en) | Method and apparatus of realizing resource provisioning | |
| CN104598776B (en) | The method and device that a kind of pair of software is tested | |
| CN103136098B (en) | The methods, devices and systems of fuzz testing | |
| CA2741982C (en) | Apparatus and method for monitoring a computer system | |
| CN106776243B (en) | Monitoring method and device for monitoring software | |
| KR20160000758A (en) | Fault Injection testing apparatus and method | |
| CN101859273A (en) | Method and device for generating test cases | |
| CN112241350B (en) | Micro-service evaluation method and device, computing device and micro-service detection system | |
| CN106407102B (en) | Fuzzy testing method, device and system for application program | |
| Zhou et al. | Logsayer: Log pattern-driven cloud component anomaly diagnosis with machine learning | |
| CN106708727B (en) | Distributed virus characteristic sample verification method and system | |
| CN107590389B (en) | Security testing method and device, electronic equipment and computer storage medium | |
| CN109670316A (en) | A kind of leak detection method and device based on Java exception | |
| CN103731315A (en) | Server failure detecting method | |
| CN110417621B (en) | Method for detecting abnormal operation state of lightweight embedded system | |
| CN113849388A (en) | Test method and device, electronic equipment and storage medium | |
| CN112860509A (en) | Dial testing alarm method and device | |
| AU2014200806B1 (en) | Adaptive fault diagnosis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |