System for testing embedded systems.
Field of the Invention
This invention relates to a new and more efficient testing system, a system to control the functionality of computers with program instructions that are built into machines to control them.
Description of the Prior Art
To achieve this functionality, the embedded computer, called the controller, recieves program instructions. These program instructions need to be carefully written to achieve the required functionality. To be able to control that all program instructions have been correctly entered into the controller, you let the controller with its program instructions run several functionality tests before delivery. Usually these tests take a lot of time to perform as there are so many functions in the controller that need to be tested. For every test that is run, the controller has to get input, output needs to be read from the controller and this has to be compared to correct output, and finally a result report has to be written.
Because of this, testing is often automised. Normal procedure to achieve this is to have someone write test cases using program instructions. The test cases are compiled and linked with the real program that is developed. Then they are together transferred to the memory of the controller to run all the test cases.
A common problem here is that the controller program grows as it is developed, and uses more and more of the memory of the controller. The test cases that do not fit in the memory are removed. Eventually there are so few test cases left, that the testing will not be good enough to find functionality faults. So far, the solution to this has been to add memory to the controller, which will make the product more expensive.
Embedded systems are computers embedded in a machine to control it. They are delivered with the machine. Examples of machines with an embedded controller are dish-washers, washing machines, mobile phones, cameras, car washes and cards playing a melody. The controller differs from a home PC, having a limited set of already known functions or tasks. Because of this the power of the processor and the internal memory are usually limited to lower the total cost of the product. This makes the space for the program instructions of the test cases very small.
Program instructions are usually written in a high level language, for example C, C++ or JAVA. To make the computer able to execute them they need to be translated into object code or machine code. This translation is called compiling, and the program that recieves code written in a high level language, called source code, and translates it into object code, is called a compiler.
Larger programs usually consist of modules. These modules can be compiled one at a time. To make the modules work together they have to be linked, which means that the modules have to have common specified addresses in the memory of the computer to be able to exchange information.
During the development phase of a program, there are usually some errors in the source code, so called bugs. Bugs are logical errors, and when the object code (compiled from the source code) is executed on the controller, it gives faulty output so the controlled machine behavies in an undesirable way.
It is very expensive to have to correct bugs discovered after the release of the controlled machine when it is assumed to do its work. American studies show that it is 100 to 1000 times more expensive than to correct the bugs during the development phase.
Modularity also gives the opportunity to test modules one at a time. Doing this decreases the testing needs as the possible cases to test get fewer.
Before accepting the source code of a module it is tested with a large amount of test cases, so that all possible combinations of input data gives the required combinations of output. As the test cases are many, it makes the testprogram instructions many too, and because of this they need a lot of space in the memory of the computer.
Summary of the Invention
The goal of this invention is to have a new more efficient system for testing of controllers and their program instructions.
The system makes it possible to do more tests without the need for the controller to have a larger memory than necessary for what it is produced to do. The program code can be tested in modules, reducing the need for testing. Tests can be automated, which reduces the time needed.
This is achieved as the controller is loaded with its compiled and linked program. Test cases are compiled one at a time, all being able to link to the program of the controller. This is done in the test computer or another computer and then transferred to the controller. The test computer links and downloads the first test case to the controller and orders it to do the test. The results of the first test case is read and reported. Then the first test case is deleted from the controller, and the next test case is downloaded, replacing the previous, and this procedure is done until all test cases are run through, hi this way a large amount of test cases can be executed without the need for extra memory in the controller.
This method to do the testing is also included in the invention.
Brief description of the Drawings
Figure 1. An example of a flow chart for testing of a controller with a control program. (l)Test computer
(2)Source code for the real program (3)Compiler
(4)Objekt code for the real program (5)Linker
(6)Final version of real program (7)Communication module (8)Embedded system
(9)Source code for the test programs
(10) Obj ekt code for the test programs
(11) Linker and controller of test execution
(12) Testprogram, compiled and linked
Detailed Description of the Invention
The testing system consists of the embedded system (8) that runs the real program (6) that should be tested, a test computer (1), a communication link (7) between the two computers and a set of program instructions (6) & (12).
The task of the test computer is to adapt the test cases for the program instructions of the embedded system, communicate with the embedded system, check the test results and report the test results.
The real program of the embedded system is compiled and linked to be executed. The test cases are also compiled. The test case that is run is in the memory of the embedded system and is linked to the real program. The next test case is adapted for the embedded system with its real program linking the testcode in such a way that it can replace the previous test case in the embedded system. During the test there is always at least one compiled and linked test case in the embedded system or in its memory. The test cases are adapted to be able to replace each other in the controller. They are linked to be able to give input and read output from the
control program at the defined addresses in the memory of the embedded system. As the different test cases are written to test different parts of the functionality of the embedded system different input and output can be used for the different test cases which means that different addresses can be used. Because of this, the test cases are not always linked to the same addresses in the memory of the embedded system, but they are still adapted not to leave any harmful code in the embedded system when replaced. There is no problem linking the test cases on another computer than the embedded system (for example on the test computer (1)) so that the controller does not have to do more work than necessary.
The embedded system has to have some space left in memory (so called excess memory) as the real program should not use every bit of its memory. The advantage is that the need for excess memory is much less than if all the test cases would have had to be downloaded at the same time.
The possibility to link several test cases to the same real program is an important feature in this invention, and has been difficult to solve. The key is to make the linking in two steps. First the real program is compiled and linked (without test cases). Not until then the test cases are linked to the real program. The method used is to produce a report at the compilation and linking of the real program, which shows where in the embedded system memory the different input and output data from the control program will be placed. This information is used as input not just for the first test case but for all test cases, and makes this special kind of linking work, that is needed in this method.
The linking is one of the two important demands making the test cases able to replace each other in the memory of the controller. The second is not to leave any harmful code from the previous test case when it is replaced with the next. This can be done in several ways. One way is to delete or overwrite references of the earlier code so that it will be inactive. Another way is to overwrite the code of the previous test case with the code of the next test case so that none of the previous code is left. It can also be possible to use part of the previous code as a module in the next test case. That kind of reused module could be a test administration module that for example starts and ends test cases.
The programming of the test cases is often done in modules. Especially functions that are needed often, so called help functions, can be useful to put in a module. The programmer instead of writing the code several times, will call the module instead to execute the function. This will make the programming more efficient as less code needs to be written, and with that the risk for errors deceases aswell. Data structures, parts of information like an account number or a string that is used often, so called help data, can be treated the same way. These modules and data structures also have to be linked with the test case and sent to the embedded system with it.
According to the invention, the test computer can have several other tasks except handling tests. It can be used for the compiling of the test cases and the real program for example. It could also include a so called editor to write the source code in and an emulator to testrun the control program on the test computer.
According to the invention the testing can be done for the whole of the real program or on separate modules of the real program. Doing it on modules makes the testing more efficient, as the amount of cases needing testing decreases. To describe the cause of this, tliink of a program consisting of three modules where every module can have 1000 different states. If the whole program consists of three modules, all states for all modules have to be combined. This will result in 1000 x 1000 x 1000 = 1000 000 000 different cases to be tested when you want to be sure of the functionality. Instead, if every module is tested on its own, there will be 1000 + 1000 + 1000 = 3000 different cases to test. Modules are tested many times during the development of the program code. Therefore it is important to minimize the amount of work needed to do the testing. Naturally the whole program has to be tested when all modules are finished, but when you know that every module works correctly, the amount of errors at this test are much fewer, and the test does not have to be run as many times as it would have had to be if you had only been testing the whole program.
Though the testing of modules very much decreases the amount of test cases, there are still many to do. The sum of all these test cases consists of a lot of
program code. There is not room for all test cases in the limited memory of the embedded system. To increase the memory of the embedded system only to be able to run tests while doing the development of the embedded system is unnecessarily expensive. This extra memory is not needed during normal execution. The invention makes it possible to do thourough testing without increasing the memory of the controller.
Using the method according to this invention makes the embedded program with its program instructions possible to test even if the machine where it should be embedded is not available. This is an advantage as you then can develop the machine and the embedded system in parallel. Also, it could be dangerous to test the embedded system while it is controlling its machine if the machine with faulty functions could harm anyone or anything.
For anything to be able to communicate with the embedded system it will need a communication module. The task of the communication module is to receive data from any protocol and translate it to data that the embedded system can understand and vice versa. The communication module normally consists of hardware (electrical circuits and wires) that can receive electrical signals, and of program instructions in the controller that decide what to do with the signals. The advantage with the communication module being able to communicate using a standard protocol is that many computers have ports for this. This makes it easier to find a computer that will work as a test computer. There are several standard communication protocols that can do the job of transferring test cases. One is RS232, an asynchronous serial protocol, which is the one expecting the least from the controller. Another type of protocols are the IETF standardised TCP/IP suite of protocols used on the Internet, and makes the controller able to communicate on long distance. USB (Universal Serial Bus) standardised by a group of companies, is today one of the most usual serial protocol for short distances, and you can find it on most computers. All of these communication standards are under constant development, and new editions are released now and then.
How to test using the method in this invention is here described using an example:
The programmer writes source code (2), compiles it(3), links it (5) and sends it to the controller (7). Using one method for writing testable code, the programmer writes the test cases (9) consisting of program instructions to check the real program (6) for a defined set of input before starting writing the source code (2). At compilation and linking of the real program (2) a report is generated where the addresses for input and output are found. These are used to link the test cases as they need this information to know where in the memory of the controller they will get their input and read the output. Now the programmer can link the first test case and transfer it (11) to the embedded system using the communication module (7), ran the first test case, read the result and report the outcome. After finishing this he will link and transfer the next test case to the controller so that it replaces the previous, and so he repeats the procedure until all test cases are run through. The whole proceeding can be automated with a so called test controller (11), making the test computer run all the tests one after the other and afterwards writing a final report of all test cases. Every test case is written to replace the earlier test case in the memory of the embedded system. If there is enough extra excess memory two or more test cases can be combined into one test case.
Above are some examples of implementations described. Naturally there can be combinations and different variations of these. The invention is not limited to the above description, but can be varied and combined within the field of the demands.