WO2017142392A1 - A system and a method to rate a software - Google Patents
A system and a method to rate a software Download PDFInfo
- Publication number
- WO2017142392A1 WO2017142392A1 PCT/MY2016/050083 MY2016050083W WO2017142392A1 WO 2017142392 A1 WO2017142392 A1 WO 2017142392A1 MY 2016050083 W MY2016050083 W MY 2016050083W WO 2017142392 A1 WO2017142392 A1 WO 2017142392A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- test
- score
- strategies
- software
- weightage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Prevention of errors by analysis, debugging or testing of software
- G06F11/3668—Testing of software
- G06F11/3672—Test management
Definitions
- the score for each of the plurality of test strategies is, but not limited to, a total score or a partial score.
- the TR module configured to perform such as, but not limited to, regulate the score for each of the plurality of test strategies with the weightage of plurality of test strategies, calculate final score and assign rating.
- the method to rate the software comprising the steps of retrieving request to rate the software, characterized in that, obtaining weightage of plurality of test strategies, identifying type of software rating to be computed, calculating score for each of the plurality of test strategies, regulating the score for each of the plurality of test strategies with the weightage of plurality of test strategies, calculating final score and assigning a final software rating.
- the step of calculating score for plurality of test case results of the plurality of test strategies further comprising the steps of assigning highest weightage for first test iteration across the plurality of test case results of the plurality of test strategies and lower weightage for subsequent test iterations based on a number of completed test iterations and assigning highest weightage for last test iteration across the plurality of test case results of the plurality of test strategies and lower weightage for preceding test iterations based on the number of completed test iterations.
- the step of calculating score for plurality of test case results of the plurality of test strategies further comprising a step of combining the scores with priority assigned to each of the plurality of test case results per the plurality of test strategies.
- the system (100) to rate the software is implemented when the software enters system testing phase i.e. after software has been built completely during coding or construction phase, it is sent to independent testing team for system testing.
- plurality of test strategies such as, but not limited to, functional test, performance test, security test and usability test are executed for the software.
- the weightage is assigned to every test strategy based on the importance of particular test strategy.
- scores are calculated based on number of test cases and test iterations either complete or incomplete. The total score for each strategy will be regulated against the strategy weightage assigned earlier. Then, the sum of all scores is calculated to get the complete score or real-time score.
- the complete or real time score is mapped with pre-defined software rating table to get the final or real-time software rating.
- the present invention determines the trustworthiness and capabilities of software under test in a form of scoring upon the completion of system test execution or on real-time based on the test strategies imposed: functional, Performance, Security and Usability. Also, the present invention provide rating to the software on the basis of multiple levels such as baseline strategies, completeness of system test execution (i.e. complete or real-time), test iterations, test case priority and test case result for each iteration.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a system (100) to rate a software comprising a test case management (TCM) module (102), a Test Strategies Identifier (TSI) module (110), a Test Iteration Identifier (TII) module (112), a Test Score Calculation (TSC) module (114) and a Test Rating (TR) module (116) to provide trustworthiness and capabilities of the software under test in a form of scoring upon the completion of system test execution or on real-time basis based on the plurality of test strategies imposed. Further, a method (200) to rate the software is provided.
Description
A SYSTEM AND A METHOD TO RATE A SOFTWARE
TECHNICAL FIELD
Embodiments of the present invention relate to software testing techniques and more particularly, to a system and a method to rate a software under test to determine trustworthiness and health of the software during system test execution.
BACKGROUND ART
Matching software system requirements with computer system's specifications is essential to ensure proper functioning of the software on the computer system. Often events such as installation of new software modules, updating, and/or customizing software modules, require running many tests in order to verify that the software system still operate as expected.
Typically, evaluation of the software products involves a smattering of benchmark tests, expert evaluations, qualitative assessments, and personal bias that may produce an inaccurate or marginally useful product evaluation or comparison. Although the selection of software products is important in developing a computing system, limited capabilities exist for rating software products.
There have been a number of solutions provided for efficient software testing and few of them discussed below:
US5731991 A describes a system for evaluating a software product. The system includes an interface, which receives product data relating to the software product, wherein a first memory stores the product data and a second memory stores a plurality of weighting values. A processor coupled to the first memory and the second memory applies a plurality of weighting values to the product data to generate at least one criterion score for the software product.
US20070282670A1 discloses a method providing a rating for a software product. The rating is obtained from a plurality of member users. A member user submits feedback data that may be stored in a database, using an interface provided on a member user terminal, the feedback data related to the software product. A member user submitting feedback data has a member user weighting, a member user weighting is preferably obtained from one or more other member users, or may be initially allocated as a default weighting.
US8355942B2A describes ranking method and system. The method includes receiving by a computing system, business requirements and associated weighting factors. The computing system receives a list of software/hardware products and associated assessment data. The business requirements are associated with product features of the software/hardware products. The computing system calculates total requirement weighting factors for the product features. The computing system stores the total requirement weighting factors. The aforesaid documents and other similar solutions may strive to provide efficient rating of software; however, they still have a number of
limitations and shortcomings such as, but not limited to, there is no unified way on giving objective and quantitative rating to the software under test once test execution is completed or on real-time basis. The various existing software certification schemes certify software by focusing on functionality, ergonomics and documentations based on ISO 25051 and ISO 9241 series (or any related software engineering standard). Since nature of the system testing is to evaluate capabilities of the software as per user expectation in environment that is as close as possible to the real environment, therefore it is significant to use system test case results as the basis to rate the software under test. Although number of defects found and fixed could serve as indicator on the goodness of the software upon the delivery, it does not translate whether features or services offered by the software are stable and trusted.
Accordingly, there remains a need in the prior art to have an improved system and method to rate a software, which overcomes the aforesaid problem and shortcomings.
Also, there remains a need in the art for a system and a method to rate a software under test upon completion of system test execution or real-time based on test strategies imposed, and to assess stability of the features of the software on the basis of their corresponding test cases and test strategies.
SUMMARY OF THE INVENTION
Embodiments of the present invention aim to provide a system and a method to rate a software by employing complete or real-time testing based on system test cases results.
In accordance with an embodiment of the present invention, the system to rate the software comprising a Test Case Management (TCM) module configured to store weightage of plurality of test strategies, plurality of test case results and priority assigned to each of the plurality of test case results per plurality of test strategies, characterized in that, a Test Strategies Identifier (TSI) module configured to obtain information from the TCM module, a Test Iteration Identifier (Til) module configured to determine information from the TCM module, a Test Score Calculation (TSC) module and a Test Rating (TR) module. In accordance with an embodiment of the present invention, the TSI module configured to obtain the information from the TCM module such as, but not limited to, type of test strategy associated with each of the plurality of test case results and weightage of the plurality of test strategies. In accordance with an embodiment of the present invention, the weightage of the plurality of test strategies is weightage of plurality of baseline test strategies agreed prior to software testing.
In accordance with an embodiment of the present invention, the Til module configured to determine the information from the TCM module such as, but not limited to, completeness of the plurality of test strategies, number of test iterations involved for each of the plurality of test case results per plurality of test strategies and the priority assigned to each of the plurality of test case results per plurality of test strategies.
In accordance with an embodiment of the present invention, the Til module further configured to determine completeness of the number of test iterations. In accordance with an embodiment of the present invention, the TSC module configured to perform such as, but not limited to, preparation of a set of calculation rules based on the number of test iterations and the priority assigned to each of the plurality of test case results per plurality of test strategies, determine score for each of the plurality of test case results per plurality of test strategies and regulate the score with the priority assigned to each of the plurality of test case results per plurality of test strategies and calculate score for each of the plurality of test strategies.
In accordance with an embodiment of the present invention, the set of calculation rules further comprises weightage for each number of test iterations and a fix base value.
In accordance with an embodiment of the present invention, the score for each of the plurality of test strategies is, but not limited to, a total score or a partial score.
In accordance with an embodiment of the present invention, the final score is, but not limited to, a complete score or a real-time score.
In accordance with an embodiment of the present invention, the TR module configured to perform such as, but not limited to, regulate the score for
each of the plurality of test strategies with the weightage of plurality of test strategies, calculate final score and assign rating.
In accordance with an embodiment of the present invention, the rating is, but not limited to, a complete rating based on a pre-defined software rating table or a real-time rating based on a pre-defined software rating table.
In accordance with an embodiment of the present invention, the method to rate the software, comprising the steps of retrieving request to rate the software, characterized in that, obtaining weightage of plurality of test strategies, identifying type of software rating to be computed, calculating score for each of the plurality of test strategies, regulating the score for each of the plurality of test strategies with the weightage of plurality of test strategies, calculating final score and assigning a final software rating.
In accordance with an embodiment of the present invention, the plurality of test strategies are, but not limited to, functional test, performance test, security test and usability test. In accordance with an embodiment of the present invention, the step of calculating score for each of the plurality of test strategies comprising a step of calculating score for plurality of test case results of the plurality of test strategies based on plurality of test iterations of the each of plurality of test case results. In accordance with an embodiment of the present invention, the step of calculating score for plurality of test case results of the plurality of test strategies
comprising the steps of assigning different weightage for each test iteration per the plurality of test case results, assigning a fix weightage value for a particular test iteration, in which weightage value for one test iteration of the plurality of test case results is required to be same with other plurality of test case results, assigning total weightage for the plurality of test case results which is same with the other plurality of test case results and assigning single fix value of base point for score calculation.
In accordance with an embodiment of the present invention, the step of calculating score for plurality of test case results of the plurality of test strategies further comprising the steps of assigning highest weightage for first test iteration across the plurality of test case results of the plurality of test strategies and lower weightage for subsequent test iterations based on a number of completed test iterations and assigning highest weightage for last test iteration across the plurality of test case results of the plurality of test strategies and lower weightage for preceding test iterations based on the number of completed test iterations.
In accordance with an embodiment of the present invention, the step of calculating score for plurality of test case results of the plurality of test strategies further comprising a step of combining the scores with priority assigned to each of the plurality of test case results per the plurality of test strategies.
In accordance with an embodiment of the present invention, the step of assigning the final software rating comprising the steps of mapping the final score with a pre-defined software rating table and assigning rating to the
software based on results of the step of mapping.
BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS
So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawing illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments.
These and other features, benefits and advantages of the present invention will become apparent by reference to the following text figure, with like reference numbers referring to like structures across the views, wherein:
Fig. 1 illustrates a system to rate a software in accordance with an embodiment of the present invention;
Fig. 2 is a process flow diagram illustrating a method to rate the software in accordance with an embodiment of the present invention;
Fig. 3 is a flow chart illustrating a step of calculating score for each of the plurality of test strategies of the method to rate the software in accordance with an embodiment of the present invention;
Figs. 4a, 4b and 4c are illustrating an example for calculating score for each of the plurality of test strategies of Figure 3;
Fig. 5 is illustrating an example of a step of regulating a final complete score of the method to rate the software in accordance with an embodiment of the present invention;
Fig. 6 is illustrating examples of a step of calculating final real-time score of the method to rate the software in accordance with an embodiment of the present invention;
Fig. 7 is a flow chart illustrating a step of assigning a final software rating of the method to rate the software in accordance with an embodiment of the present invention;
Fig. 8 is illustrating an example of the step of assigning the final software rating of the method to rate the software in accordance with an embodiment of the present invention; Fig. 9 is a flow chart illustrating a process for determining completeness of plurality of test strategies in accordance with an embodiment of the present invention;
Fig. 10 is illustrating an example of calculating partial scores for complete as well as incomplete plurality of test strategies in the process of determine completeness of plurality of test strategies of Figure 9;
Fig. 11 is illustrating an example of calculating partial scores based on completed test iterations in the process of determine completeness of plurality of test strategies of Figure 9. DETAILED DESCRIPTION OF THE PREFFERED EMBODIMENTS
While the present invention is described herein by way of example using embodiments and illustrative drawings, those skilled in the art will recognize that the invention is not limited to the embodiments of drawing or drawings described, and are not intended to represent the scale of the various components. Further, some components that may form a part of the invention may not be illustrated in certain figures, for ease of illustration, and such omissions do not limit the embodiments outlined in any way. It should be understood that the drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the scope of the present invention as defined by the appended claim. As used throughout this description, the word "may" is used in a permissive sense (i.e. meaning having the potential to), rather than the mandatory sense (i.e. meaning must). Further, the words "a" or "an" mean "at least one" and the word "plurality" means "one or more" unless otherwise mentioned. Furthermore, the terminology and phraseology used herein is solely used for descriptive purposes and should not be construed as limiting in scope. Language such as "including," "comprising," "having," "containing," or "involving," and variations thereof, is intended to be broad and encompass the subject matter listed thereafter, equivalents, and additional subject matter not recited, and is not intended to exclude other additives, components, integers or steps. Likewise,
the term "comprising" is considered synonymous with the terms "including" or "containing" for applicable legal purposes. Any discussion of documents, acts, materials, devices, articles and the like is included in the specification solely for the purpose of providing a context for the present invention. It is not suggested or represented that any or all of these matters form part of the prior art base or were common general knowledge in the field relevant to the present invention.
In this disclosure, whenever a composition or an element or a group of elements is preceded with the transitional phrase "comprising", it is understood that we also contemplate the same composition, element or group of elements with transitional phrases "consisting of", "consisting", "selected from the group of consisting of, "including", or "is" preceding the recitation of the composition, element or group of elements and vice versa. Referring to the drawings, the invention will now be described in more detail. In accordance with an embodiment of the present invention, a system (100) to rate a software, as shown in Figure 1 , comprising a Test Case Management (TCM) module (102), a Test Strategies Identifier (TSI) module (110), a Test Iteration Identifier (Til) module (112), a Test Score Calculation (TSC) module (114) and a Test Rating (TR) module (116).
In accordance with an embodiment of the present invention, the TCM module (102) is configured to store weightage (104) of plurality of test strategies, plurality of test case results (106), priority assigned (108) to each of the plurality of test case results per plurality of test strategies. The weightage (104) of plurality of test strategies is, but not limited to, weightage of baseline test
strategies agreed prior to software testing. Further, the plurality of test strategies are, but not limited to, functional test, performance test, security test and usability test. Also, the plurality of test case results (106) are set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly.
In accordance with an embodiment of the present invention, the TSI module (110) is configured to obtain various information from the TCM module (102) such as, but not limited to, type of test strategy associated with each of the plurality of test case results (106) and weightage (104) of the plurality of test strategies.
In accordance with an embodiment of the present invention, the Til module (112) is configured to determine number of information from the TCM module (102), besides TSI module (110), such as, but not limited to, completeness of the plurality of test strategies, number of test iterations involved for each of the plurality of test case results (106) per plurality of test strategies and the priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies. Further, the Til module (112) is configured to determine completeness of the number of test iterations.
In accordance with an embodiment of the present invention, the TSC module (114) is configured to prepare a set of calculation rules. The set of calculation rules are based on the number of test iterations and the priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies. Further, the set of calculation rules comprises weightage for
each number of test iterations and a fix value base. Also, the TSC module (114) is configured to determine score for each of the plurality of test case results (106) per plurality of test strategies and regulate the score with the priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies. Thereafter, the TSC module (114) calculates the score for each of the plurality of test strategies. The score for each of the plurality of test strategies is, but not limited to, a total score or a partial score.
In accordance with an embodiment of the present invention, the TR module (116) is configured to regulate the score for each of the plurality of test strategies with the weightage (104) of plurality of test strategies. Thereafter, the TR module (116) calculates the final score and assigns the rating to the software. Preferably, the final score is, but not limited to, a complete score or a real-time score. Further, the rating is, but not limited to, a complete rating or a real-time rating and the rating is assigned preferably based on a pre-defined software rating table.
As shown in Figure 1 , the system (100) to rate the software is implemented when the software enters system testing phase i.e. after software has been built completely during coding or construction phase, it is sent to independent testing team for system testing. During system testing, plurality of test strategies such as, but not limited to, functional test, performance test, security test and usability test are executed for the software. Further, the weightage is assigned to every test strategy based on the importance of particular test strategy. Thereafter, for every test strategy, scores are calculated based on number of test cases and test iterations either complete or incomplete.
The total score for each strategy will be regulated against the strategy weightage assigned earlier. Then, the sum of all scores is calculated to get the complete score or real-time score. Finally, the complete or real time score is mapped with pre-defined software rating table to get the final or real-time software rating.
Figure 2 is a process flow diagram illustrating a method (200) to rate the software in accordance with an embodiment of the present invention. As shown in Figure 2, at step 202, a request to rate the software is received. Upon receiving the request, the weightage (104) of plurality of test strategies is obtained at step 204 and are stored in the test case management (TCM) module (102). Also, plurality of test case results (106) and priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies are stored in the TCM module (102). Further, the weightage (104) of plurality of test strategies is, but not limited to, weightage of baseline test strategies agreed prior to software testing. Also, the plurality of test strategies are, but not limited to, functional test, performance test, security test and usability test. After obtaining the weightage (104) of the plurality of strategies, at step 206, desired type of software rating is identified. Preferably, the type of software rating is, but not limited to, a complete rating or a real-time rating. The test strategies identifier (TSI) module (110) performs both, the steps 204 and 206.
At step 208, score for each of the plurality of strategies such as, but not limited to, functional test, performance test, security test and usability test is calculated as follows:
- if the type of software rating, at step 206, is identified as complete rating then the scores for each functional test, performance test, security test and usability test are calculated for software under test that has completed execution of system testing. Preferably, this score is a total score.
- if the type of software rating, at step 206, is identified as real-time rating then the partial scores for each functional test, performance test, security test and usability test are calculated for software under test that has on-going execution of system testing. Preferably, this score is a partial score.
Further, the step 208 is performed by the test iteration identifier (Til) module (112) and the test score calculation (TSC) module (114). Figure 3 is a flow chart (300) illustrating the step of calculating (208) score for each of the plurality of test strategies of the method (200) to rate the software in accordance with an embodiment of the present invention.
As shown in Figure 3, at step 302, the type of preferred test strategy from the plurality of test strategies i.e. functional test, performance test, security test and usability test is provided. At step 304, the type of test strategy has been identified as follows:
- if the test strategy is functional test, performance test and security test then the step 306 will be processed,
- if the test strategy is usability test then the step 318 will be processed.
At step 306, the number of plurality of test iterations involved for the test strategies is identified and segregated on the basis of the number of plurality of test iterations as follows:
- at step 308, if number of plurality of iterations is 2 then score for each of the plurality of test strategies is calculated, at step 310, by 2-points system;
- at step 308, if number of plurality of iterations is 3 then score for each of the plurality of test strategies is calculated, at step 312, by 3-points system; and
- at step 308, if number of plurality of iterations is 4 then score for each of the plurality of test strategies is calculated, at step 314, by 4-points system.
Further, at step 316, the scores calculated at steps 310, 312 and 314 are combined in accordance with the priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies to get the actual score for each of the plurality of test case results (106).
As shown in Figure 3, at step 318, the type of usability test is identified and scores for each of the type is obtained as follows:
- at step 320, if the usability test is of the user test type, user experience test score is determined at step 324; and
- at step 320, if the usability test is of the heuristic evaluation type, the score for heuristic evaluation is calculated at step 322
Thereafter, all scores from steps 31 6, 322 and 324 are summed up to get total scores for each of the plurality of test strategies i.e. for functional test, performance test, security test and usability test at step 208. Figs. 4a, 4b and 4c are illustrating an example (400) for calculating
(208) score each of the plurality of test strategies.
For example, an X-points system introduced in steps 31 0, 31 2 and 314 is based on equation (1 ):
Points = Weightage (W) χ Base point (B) (1 ) which is given for each test iteration per plurality of test case results (1 06). The base point (B) remains the same for every test iteration. The weightage (W) value differentiates the point given to each test iteration. Total full points for each of the plurality of test case results (1 06) after taking into account the number of test iterations it has to go through should be the same with other plurality of test case results (1 06).
As shown in Figure 4a, there are two options (402, 404) that could be considered:
- option 1 (402): assumption that 1 st test iteration carries highest weightage as compared to subsequent test iterations. This is due to the importance of passing each of the plurality of test case results
(106) for the first time it is executed to demonstrate the stability of the
feature being tested. Thus, the weightage for 1 st test iteration should be a fix value for all plurality of test case results (106).
- option 2 (404): assumption that last test iteration (2nd, 3rd or 4th test iteration) carries the highest weightage as compared to preceding test iterations. This is due to the need of passing each of the plurality of test case results (106) during final test iteration as part of testing exit criteria. Thus, the weightage for last test iteration should be a fix value for all the plurality of test case results (106). Further, in the above example each point for each of the plurality of test case results (106) is combined with priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies earlier. Also, the total score of all the plurality of test case results (106) is calculated against ideal score before converting to percentage.
As shown in Figure 4b, the score for option 1 (402) is calculated as follows:
If the score for functional test strategy is calculated by using above-mentioned option (402) and base point (B) 10. Since the functional test results involved 3 test iterations, thus the 3-points system is being used in this example.
3-points system is applied for each test case:
- 6B + 3B + 1 B or 6(10) + 3(10) + 1 (10) = 100 (The total points for each test case if all test iterations have PASS results is 100) - This total point is regulated with the priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies to
get the actual score for each of the plurality of test case results (106).
In case of TC1 :
- priority for TC1 is 3
- ideally, if all test iterations are PASS, the ideal score is 100 x 3 = 300
- since after test completed, TC1 passes all test iterations, therefore the actual score is 300
- similar approach is applied for remaining test cases
- once the actual scores are obtained for all test cases, all of them are summed up to get total score.
- Thus, score for Functional Test score is obtained by dividing Total Score against Ideal Score x 100%, which is 3760/4300 x 100% = 87.4% As shown in Figure 4c, the score for option 2 (404) is calculated as follows:
If the score for functional test strategy is calculated by using above-mentioned option (404) and base point (B) 10. Since the functional test results involved 3 test iterations, thus the 3-points system is being used in this example. In this scenario, it is assumed that 3rd or last iteration is the most important. Therefore, Option 2 (404) for 3-points system is applied for each test case:
- 1 B + 3B + 6B or 1 (10) + 3(10) + 6(10) = 100 (The total points for each test case if all iterations have PASS results is 100) - This total point is regulated with the priority assigned (108) to each of the plurality of test case results (106) per plurality of test strategies to
get the actual score for each of the plurality of test case results (106).
In case of TC14:
- priority for TC14 is 3
- ideally, if all test iterations are PASS, the ideal score is 100 x 3 = 300
- since after test completed, TC14 only passes 2nd and 3rd iterations, therefore the actual score is{0(10) + 3(10) + 6(10) } x 3 = 270.
- similar approach is applied for remaining test cases
- once the actual score are obtained for all test cases, all of them are summed up to get total scores.
- Thus, score for Functional Test score is obtained by dividing Total Score against Ideal Score x 100%, which is 1880/200 x 100% = 89.7%. At step 210, the score calculated at step 208 is regulated by comparing them with the weightage (104) of the plurality of strategies. This step is performed by the test rating (TR) module (116).
Figure 5 is illustrating an example (500) of the step of regulating (210) the score of the method (200) to rate the software in accordance with an embodiment of the present invention. This particular example illustrates the step of regulating final complete score of the method (200) on the basis of pre-defined weightage / percentage for each of the plurality of strategies. It is assumed that the final software rating is given based on the following agreed test strategies weightage:
Functional Test Weightage = 50% or 0.5
Performance Test Weightage = 20% or 0.2
Security Test Score Weightage = 15% or 0.15
Usability Test Weightage = 15% or 0.15
Total Score = 100% or 1 .0
As shown in Figure 5, the regulated score for each of the plurality of test strategies is computed as:
It is apparent from the aforesaid example that the regulated score for each of the plurality of test strategies is directly proportional to the weightage (104) of the plurality of test strategies.
At step 212, final score has been calculated. Preferably, the final score is, but not limited to, a complete score or a real-time score. Further, the complete score is calculated for software under test that has completed execution of system testing and the real-time score is calculated for software under test that has on-going execution of system testing.
Figure 6 is illustrating examples (600, 602) of the step of calculating (212) final score of the method (200) to rate the software in accordance with an embodiment of the present invention. This particular example illustrates the step of calculating final real-time score of the method (200) on the basis of pre-defined weightage or percentage for each of the plurality of strategies. It is assumed that the final software rating is given based on the following agreed test strategies weightage:
Functional Test Weightage = 50% or 0.5
Performance Test Weightage = 20% or 0.2
Security Test Score Weightage = 15% or 0.15
Usability Test Weightage = 15% or 0.15
Total Score = 100% or 1 .0 As shown in Figure 6, the final score, real-time score in this example, for each of the plurality of test strategies, in example 600, is computed as:
As shown in Figure 6, the final score, real-time score in this example, for each of the plurality of test strategies, in example 602, is computed as:
Test Strategy Regulated Score
Functional Test Weightage = 16.20
Performance Test Weightage = -
Security Test Score Weightage = -
Usability Test Weightage = -
Total (Real-time) Score = 16.20
In view of the above-mentioned examples, it is apparent that the real-time score for each of the plurality of test strategies can be obtain and health of the software during on-going execution can be analyzed.
At step 214, final software rating has been assigned to the software on the basis of, but not limited to, a pre-defined software rating table. Further, the final software rating is, but not limited to, a full rating or a partial rating. Figure 7 is a flow chart illustrating the step of assigning (214) the final software rating of the method (200) to rate the software in accordance with an embodiment of the present invention. As shown in Figure 7, at step 2142, mapping of the final score calculated at the step 212 is performed with the pre-defined software rating table and the rating is assigned to the software under test, at step 2144, on the basis of said mapping scores. The software is either the one has completed test execution or in-progress.
Figure 8 is illustrating an example (700) of the step of assigning (214) the final software rating of the method (200) to rate the software in accordance with an embodiment of the present invention. As shown in Figure 8, in this
specific example star-based rating has been assigned to the software. However, the rating can be of form such as, but not limited to, grade-based (i.e. Grade A, Grade B, Grade C, Grade D and Grade E), categorical-based (i.e. Excellent, Good, Fair, Poor and Very Poor) etc.
For an example, final software rating for the example discussed in Figure 5 would be 5-stars as its total score is 89.4%, whereas final software rating for the examples discussed in Figure 6 would be 3-stars for 46.85% real-time score and 2-stars for 16.20% real-time score.
Figure 9 is a flow chart illustrating the process (800) of determines completeness of plurality of test strategies in accordance with an embodiment of the present invention. As shown in Figure 9, at step 802, current state of incomplete test or test-in-progress is assessed.
As shown in Figure 9, at step 804, if there is any completed strategy out of the plurality of test strategies then step 806 processed. At step 806, the completed test strategy out of the plurality of test strategies is identified. Thereafter, at step 808, if the completed test strategy involves any incomplete test iteration (or test iteration is in progress), partial scores are calculated for mix of completed and incomplete test iterations in the completed test strategy at step 810. If, at step 808, the completed test strategy involves full complete test iterations, partial scores are calculated for the completed test iterations at step 812.
As shown in Figure 9, at step 804, if there is no completed strategy out of the plurality of test strategies then step 814 processed. At step 814, test iteration in which the plurality of test case results are currently executed is identified. Thereafter, at step 816, partial scores are calculated based on completed test iterations.
Figure 10 is illustrating an example (900) of calculating partial scores for complete as well as incomplete plurality of test strategies in the process of determine completeness of plurality of test strategies of Figure 9. The Figure 10 represents a scenario where test execution is currently in progress where functional testing has been fully completed while performance testing is still on-going. Therefore, the steps to calculate real-time score calculation is used.
It assumed that the total score for functional test is 89.7%. The performance test requires 3 iterations to complete but is currently in progress. As of current status, performance test execution has completed test Iteration 1 but waiting to start test Iteration 2. Thus, the real-time score is obtained as follows:
- 1 B + 3B + 6B (with 10 as base point) option is used. Then, it is regulated with test case priority.
- using TC3 as example, the actual score for TC3 is {1 (10) + 3(0) + 6(0)} x 1 = 10 (the ideal score would be 100).
- same step is repeated for the remaining test cases.
- total score for completed test Iteration 1 would be 120 while the total ideal score would be 1200. Thus, the real-time score for performance test is 120/1200 x 100% = 10%
Figure 11 is illustrating an example (902) of calculating partial scores based on completed test iterations in the process of determine completeness of plurality of test strategies of figure 9. The Figure 11 represents a scenario where none of the test strategies are completed. In this scenario, functional test execution is currently in progress where test iteration 1 is completed while test iteration 2 is still on going. Thus, the steps to calculate real-time score calculation is used.
It is assumed that some test cases have progressed to test Iteration 2 while others only completed test Iteration 1 . Last iteration becomes the most important iteration, thus 1 B + 3B + 6B applies (with 10 as base point) option is used.
As shown in Figure 11 , for test cases that has completed up to test Iteration 2, the following calculation applies:
- using TC2 as example, the calculation would be {1 (10) + 3(10) + 6(0)} x 3 = 120
- the ideal score for TC2 should be 300
- same approach is used for other test cases that have completed up to test Iteration 2
As shown in Figure 11 , for test cases that has completed up to test Iteration 1 only, the following calculation applies:
using TC10 as example, the calculation would be {1 (10) + 3(0) + 6(0)} x 2 = 20
the ideal score for TC2 should be 200
same approach is used for other test cases that have completed up to only test Iteration 1
the real-time score would be 680/2100 x 100% = 32.4% The above-mentioned system and method to rate the software overcomes the problems and shortcomings of the existing software rating/testing systems and methods and provides numerous advantages over them. The present invention provides an objective assessment on the software quality attributes and capabilities, both functional and non-functional towards giving a proper rating to the software capabilities by using test case execution results. This enables assessing the stability and reliability of software features built by developers via the corresponding test cases and besides, it helps to strengthen the software confidence among the end-users, clients or recipients on the capabilities of the software delivered to them. Further, the disclosed system and method to rate the software provides the rating by utilizing the data from existing TCM module. The present invention determines the trustworthiness and capabilities of software under test in a form of scoring upon the completion of system test execution or on real-time based on the test strategies imposed: functional, Performance, Security and Usability. Also, the present invention provide rating to the software on the basis of multiple levels such as baseline strategies, completeness of system test execution (i.e. complete or real-time), test iterations, test case priority and test case result for each iteration. Various modifications to these embodiments are apparent to those skilled in the art from the description and the accompanying drawings. The
principles associated with the various embodiments described herein may be applied to other embodiments. Therefore, the description is not intended to be limited to the embodiments shown along with the accompanying drawings but is to be providing broadest scope of consistent with the principles and the novel and inventive features disclosed or suggested herein. Accordingly, the invention is anticipated to hold on to all other such alternatives, modifications, and variations that fall within the scope of the present invention and appended claim.
Claims
1 . A system (100) to rate a software comprising:
a Test Case Management (TCM) module (102) configured to store weightage (104) of plurality of test strategies, plurality of test case results (106) and priority assigned (108) to each of said plurality of test case results (106) per plurality of test strategies;
characterized in that:
a Test Strategies Identifier (TSI) module (110) configured to obtain information from said TCM module (102);
a Test Iteration Identifier (Til) module (112) configured to determine information from said TCM module (102);
a Test Score Calculation (TSC) module (114); and
a Test Rating (TR) module (116).
2. The system (100) as claimed in claim 1 , wherein said TSI module (110) is configured to obtain the following information from said TCM module (102):
- type of test strategy associated with each of said plurality of test case results (106); and
- weightage (104) of said plurality of test strategies.
3. The system (100) as claimed in claim 2, wherein said weightage (104) of said plurality of test strategies is weightage of plurality of baseline test strategies agreed prior to software testing.
4. The system (100) as claimed in claim 1 , wherein said Til module (112) is configured to determine the following information from said TCM module (102):
- completeness of said plurality of test strategies;
- number of test iterations involved for each of said plurality of test case results (106) per plurality of test strategies; and
- said priority assigned (104) to each of said plurality of test case results per plurality of test strategies.
5. The system (100) as claimed in claim 4, wherein said Til module (112) further configured to determine completeness of said number of test iterations.
6. The system (100) as claimed in claim 1 , wherein said TSC module (114) is configured to perform the following:
- preparation of a set of calculation rules based on said number of test iterations and said priority assigned (108) to each of said plurality of test case results (106) per plurality of test strategies; and
- determine score for each of said plurality of test case results (106) per plurality of test strategies and regulate said score with said priority assigned (108) to each of said plurality of test case results (106) per plurality of test strategies; and
- calculate score for each of said plurality of test strategies.
7. The system (100) as claimed in claim 6, wherein said set of calculation rules further comprises weightage for each number of test iterations and a fix base value.
8. The system (100) as claimed in claim 6, wherein said score for each of said plurality of test strategies is a total score.
9. The system (100) as claimed in claim 6, wherein said score for each of said plurality of test strategies is a partial score.
10. The system (100) as claimed in claim 1 , wherein said TR module (116) is configured to perform the following:
- regulate said score for each of said plurality of test strategies with said weightage (104) of plurality of test strategies;
- calculate final score; and
- assign rating.
11 . The system (100) as claimed in claim 10, wherein said final score is a complete score.
12. The system (100) as claimed in claim 10, wherein said final score is a real-time score.
13. The system (100) as claimed in claim 10, wherein said rating is a complete rating based on a pre-defined software rating table.
14. The system (100) as claimed in claim 10, wherein said rating is a real-time rating based on a pre-defined software rating table.
15. A method (200) to rate a software, comprising the steps of:
retrieving (202) request to rate said software;
characterized in that
obtaining (204) weightage (104) of plurality of test strategies;
identifying (206) type of software rating to be computed;
calculating (208) score for each of said plurality of test strategies;
regulating (210) said score for each of said plurality of test strategies with said weightage (104) of plurality of test strategies;
calculating (212) final score; and
assigning (214) a final software rating.
16. The method (200) as claimed in claim 1 5, wherein said plurality of test strategies are functional test, performance test, security test and usability test.
17. The method (200) as claimed in claim 15, wherein said step of calculating (208) score for each of said plurality of test strategies comprising a step of calculating score for plurality of test case results (106) of said plurality of test strategies based on plurality of test iterations of said each of plurality of test case results (106).
18. The method (200) as claimed in claim 15, wherein said step of calculating (208) score for plurality of test case results (106) of said plurality of test strategies comprising the steps of:
assigning different weightage for each test iteration per said plurality of test case results (106);
assigning highest weightage for first test iteration across said plurality of
test case results (106) of said plurality of test strategies and lower weightage for subsequent test iterations based on a number of completed test iterations; assigning highest weightage for last test iteration across said plurality of test case results (106) of said plurality of test strategies and lower weightage for preceding test iterations based on said number of completed test iterations; assigning a fix weightage value for a particular test iteration in which weightage value for one test iteration of said plurality of test case results (106) is required to be same with other plurality of test case results (106);
assigning total weightage for said plurality of test case results (106) which is same with said other plurality of test case results (106); and
assigning single fix value of base point for score calculation.
19. The method (200) as claimed in claim 18, wherein said step of calculating (208) score for plurality of test case results (106) of said plurality of test strategies further comprising a step of combining said scores with priority assigned (108) to each of said plurality of test case results (106) per said plurality of test strategies.
20. The method (200) as claimed in claim 15, wherein said step of assigning (214) said final software rating comprising the steps of:
mapping (2142) said final score with a pre-defined software rating table; and
assigning (2144) rating to said software based on results of said step of mapping (2142).
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| MYPI2016000280 | 2016-02-17 | ||
| MYPI2016000280 | 2016-02-17 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2017142392A1 true WO2017142392A1 (en) | 2017-08-24 |
Family
ID=59625364
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/MY2016/050083 Ceased WO2017142392A1 (en) | 2016-02-17 | 2016-11-30 | A system and a method to rate a software |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2017142392A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108711008A (en) * | 2018-05-17 | 2018-10-26 | 北京中软国际信息技术有限公司 | A kind of various dimensions software project health degree appraisal procedure based on convolutional neural networks |
| CN110955593A (en) * | 2019-10-28 | 2020-04-03 | 北京三快在线科技有限公司 | Client test method and device, electronic equipment and readable storage medium |
| WO2022042726A1 (en) * | 2020-08-28 | 2022-03-03 | 中兴通讯股份有限公司 | System testing method and apparatus, device, and storage medium |
| CN117194267A (en) * | 2023-09-26 | 2023-12-08 | 江苏天好富兴数据技术有限公司 | Software quality rating system based on cloud platform |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1526459A2 (en) * | 2003-10-23 | 2005-04-27 | Agilent Technologies, Inc. | Improved method for diagnosing complex system faults |
| US20070006161A1 (en) * | 2005-06-02 | 2007-01-04 | Kuester Anthony E | Methods and systems for evaluating the compliance of software to a quality benchmark |
| US20090287679A1 (en) * | 2008-05-14 | 2009-11-19 | International Business Machines Corporation | Evaluation of tamper resistant software system implementations |
| US20110246834A1 (en) * | 2010-03-31 | 2011-10-06 | Microsoft Corporation | Testing software in electronic devices |
| US20130159774A1 (en) * | 2011-12-19 | 2013-06-20 | Siemens Corporation | Dynamic reprioritization of test cases during test execution |
-
2016
- 2016-11-30 WO PCT/MY2016/050083 patent/WO2017142392A1/en not_active Ceased
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1526459A2 (en) * | 2003-10-23 | 2005-04-27 | Agilent Technologies, Inc. | Improved method for diagnosing complex system faults |
| US20070006161A1 (en) * | 2005-06-02 | 2007-01-04 | Kuester Anthony E | Methods and systems for evaluating the compliance of software to a quality benchmark |
| US20090287679A1 (en) * | 2008-05-14 | 2009-11-19 | International Business Machines Corporation | Evaluation of tamper resistant software system implementations |
| US20110246834A1 (en) * | 2010-03-31 | 2011-10-06 | Microsoft Corporation | Testing software in electronic devices |
| US20130159774A1 (en) * | 2011-12-19 | 2013-06-20 | Siemens Corporation | Dynamic reprioritization of test cases during test execution |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108711008A (en) * | 2018-05-17 | 2018-10-26 | 北京中软国际信息技术有限公司 | A kind of various dimensions software project health degree appraisal procedure based on convolutional neural networks |
| CN108711008B (en) * | 2018-05-17 | 2022-03-29 | 北京中软国际信息技术有限公司 | Multi-dimensional software project health degree evaluation method based on convolutional neural network |
| CN110955593A (en) * | 2019-10-28 | 2020-04-03 | 北京三快在线科技有限公司 | Client test method and device, electronic equipment and readable storage medium |
| WO2022042726A1 (en) * | 2020-08-28 | 2022-03-03 | 中兴通讯股份有限公司 | System testing method and apparatus, device, and storage medium |
| CN117194267A (en) * | 2023-09-26 | 2023-12-08 | 江苏天好富兴数据技术有限公司 | Software quality rating system based on cloud platform |
| CN117194267B (en) * | 2023-09-26 | 2024-04-26 | 江苏天好富兴数据技术有限公司 | Software quality rating system based on cloud platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Freimut et al. | An industrial case study of implementing and validating defect classification for process improvement and quality management | |
| Briand et al. | A replicated assessment and comparison of common software cost modeling techniques | |
| Abdul Hameed et al. | Assessing the influence of environmental and CEO characteristics for adoption of information technology in organizations | |
| Hameed et al. | Management & Innovation | |
| Guo et al. | " Not my bug!" and other reasons for software bug report reassignments | |
| US8997052B2 (en) | Risk-based test plan construction | |
| US8412556B2 (en) | Systems and methods for facilitating an analysis of a business project | |
| Khandelwal et al. | Impact of gamification on code review process: An experimental study | |
| WO2017142392A1 (en) | A system and a method to rate a software | |
| Knauss et al. | Investigating the impact of software requirements specification quality on project success | |
| US11119763B2 (en) | Cognitive selection of software developer for software engineering task | |
| Methawachananont et al. | Software process capability self-assessment support system based on task and work product characteristics: A case study of ISO/IEC 29110 standard | |
| Šendelj et al. | Multi-criteria decision making for optimal configuration of business process model families | |
| Okunlola et al. | Determining the performance of procurement methods against selection criteria using Outranking-satisfying methodology | |
| KR20240009646A (en) | Method of calculating corporate value of startups | |
| Pushak et al. | Empirical scaling analyzer: An automated system for empirical analysis of performance scaling | |
| Gillespie | Security Compliance in Large Private Enterprise Information Systems Utilizing DevOps: An Exploratory Study | |
| JP7793738B1 (en) | Calculation device, calculation method, and program for calculating test case priority | |
| Karmito et al. | Project management: Model research in success rate of a digital start-up project | |
| Razak et al. | The information and communication technology governance maturity level for malaysian public sector | |
| Pittaras et al. | Crowdsourcing in single-document summary evaluation: the argo way | |
| Shida et al. | A Study of software estimation factors extracted using covariance structure analysis | |
| Satriani et al. | Applying Importance-Performance Analysis to Identify Strategic Factors Enhancing User Satisfaction in Dana | |
| Radeck et al. | Refining and Validating Change Requests from a Crowd to Derive Requirements | |
| Tobing et al. | Assessment system of halal product assurance implementation in Indonesian companies using ID3 and forward chaining algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 16890787 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 16890787 Country of ref document: EP Kind code of ref document: A1 |