1 Software Development ToolsCOMP220 Seb Coope Ant, Testing and JUnit Capturing test results These slides are mainly based on “Java Development with Ant” - E. Hatcher & S.Loughran. Manning Publications, 2003

2 Capturing test resultsThe task can collect test results by using formatters. One or more elements can be nested either directly under or under the (and to be discussed soon) Ant includes three types of formatters: brief, plain, xml 4.6.1

3 Capturing test resultsAnt task result formatter types type Description ================================================================== brief Provides details of each test case run with summary statistics on numbers of its Test method runs, Failures, Errors, and overall Time elapsed, and also details on each failed test method , all of this in text format If usefile="true", each test case generates its own txt file. plain Like formatter brief, and additionally the time taken by each test method run (not only by those failed) , all of this in text format . xml Like formatter plain, and additionally date/time of testing and Ant’s properties at this time, all of this in XML format If usefile="true", each test case generates its own XML file. To avoid duplication on the console of the summary statistics on test results, switch off printsummary attribute: haltonfailure="true" would restrict the information of the above formatters to the first failure. , system out, and system error output of each test case.

4 Capturing test resultsBy default, ’s output is directed to files (usefile="true"), but can also be directed to Ant’s console (usefile="false"). Update our target test-brief in mybuild.xml (see Slide14 in Part 12. Ant and JUnit) by including in : the build failure upon test failure (haltonfailure="true"), brief console output , and turned off the printsummary option to not duplicate with the output from the brief formatter COMPARE formatters brief, plain and xml by varying and running this target. Run this target in mybuild.xml.

5 Capturing test resultsH:\Antbook\ch04>ant -f mybuild.xml test-brief Buildfile: C:\Antbook\ch04\mybuild.xml [echo] Building Testing Examples test-brief: [junit] Testsuite: org.eclipseguide.persistence.FilePersistenceServicesTest [junit] Tests run: 5, Failures: 2, Errors: 0, Time elapsed: sec [junit] Testcase: testWrite(org.eclipseguide.persistence. FilePersistenceServicesTest): FAILED [junit] NOT WRITTEN??? [junit] junit.framework.AssertionFailedError: NOT WRITTEN??? [junit] at org.eclipseguide.persistence.FilePersistenceServicesTest. testWrite(Unknown Source) [junit] This produces the following output on testWrite and testRead : Our message to ourselves, if testWrite FAILED ant -f mybuild.xml test-brief (continued)

6 Capturing test results[junit] Testcase: testRead(org.eclipseguide.persistence. FilePersistenceServicesTest): FAILED [junit] expected:<[One, Two, Three]> but was: [junit] junit.framework.AssertionFailedError: expected:<[One, Two, Three]> but was: [junit] at org.eclipseguide.persistence.FilePersistenceServicesTest. testRead(Unknown Source) [junit] BUILD FAILED C:\Antbook\ch04\mybuild.xml:157: Test org.eclipseguide.persistence.FilePersistenceServicesTest failed Total time: 1 second continuation: output on testRead: As shows, assertions in testWrite and testRead methods in FilePersistenceServicesTest failed. Note that SimpleTest did not run since haltonfailure="true".

7 Capturing test resultsNow we’re getting somewhere: tests run as part of our regular build, test failures cause our build to fail: BUILD FAILED we get enough information to see what is going on. By default, formatters write their output to files either in the base directory of the build file, or in the directories specified in the or elements by their optional attribute todir. But our choice usefile="false" causes formatters to write to the Ant console instead of writing to a file. TRY also usefile="true". What will you see on console? And in the base directory C:\Antbook\ch04? todir attribute of or : Directory to write the reports to

8 Capturing test resultsAlso, we turned off the printsummary option as it duplicates and interferes with the output in the console from the brief or plain formatter. In case of usefile="true", it makes sense to turn on the printsummary (to the console). xml formatter is better to use with usefile="true". It generates a huge XML output by listing all Ant’s properties The task allows using simultaneously more than one , so you can direct results toward several formatters at a time, as in the example below. The formatter, shows only the most important information rather than line numbers tracing back into JUnit’s classes. It’s worth noting that the stack trace shown is abbreviated by the formatter, showing only the most important pieces rather than line numbers tracing back into JUnit’s classes.

9 Add this as a NEW target in mybuild.xmlXML formatter Saving the results to XML files lets you process them in a number of ways (e.g. transforming to HTML). Our testing task and target now evolves to this: name= "org.eclipseguide.persistence.FilePersistenceServicesTest"/> To avoid duplication in console of summary statistics C:\Antbook\ch04\ build\data mybuild.xml, target name=“test-xml” Add this as a NEW target in mybuild.xml

10 C:\Antbook\ch04>ant -f mybuild.xml clean test-xmlXML formatter (cont.) The effect of the above is to create an XML reports in the ${test.data.dir} (i.e., build/data) directory for each test case run. In our example, we get in build/data one (it could be more) XML file named like (actually, one line) TEST-org.eclipseguide.persistence. FilePersistenceServicesTest.xml. Note that for this to work we should use tasks in appropriate target test-init (besides the usual init target) to create all directories required for testing and test reports, including build\data for XML reports and build\reports to be used later for HTML reports. Recall that you should use properties in mybuild.xml for all directories used. TRY it: C:\Antbook\ch04>ant -f mybuild.xml clean test-xml and inspect the above xml file just generated. ant -f mybuild.xml clean test-xml Viewing System.out and System.err output???

11 Running multiple tests under You can specify any number of sub-elements in task, but that is still time consuming for writing your build file, indeed, each individual test case should explicitly be mentioned. But, instead of individual elements, you can use with a fileset containing non-fixed number of files. This includes all your test cases without mentioning any one of them individually. In this case there is no need to update the build file when adding new test cases as soon as they automatically occur in the fileset under . That is why is better to use than multiple tags. (here we do not mean the time of running it) 4.6.2 (i.e. appropriate testing .java classes into a directory) S

12 Running multiple tests under TIP: Standardize the naming scheme of your JUnit test cases for easy fileset inclusions, and possible exclusions of helper classes (or abstract test case classes). Thus, real Junit Test Cases should end with the suffix Test according to the normal convention-naming scheme. For example, FilePersistenceServicesTest and SimpleTest are our test cases, whereas Any other helper and abstract classes (possibly used by test cases) should not have the suffix Test. (i.e. testing Java classes) We use “TestCase” as the suffix for our abstract test cases.

13 Running multiple tests under Create in mybuild.xml a new target "test-batch" with using : includes="**/*Test.class"/> Directory for XML reports for each test case run Running all test cases in ${build.test.dir} at any depth

14 Running multiple tests under The includes="**/*Test.class" attribute above and our agreement on naming test cases ensures that only our concrete test cases are considered. Now TRY >ant -f mybuild.xml clean test-batch If you choose haltonfailure= "yes", then included test cases will run in some order only until one of them fails. All these really running tests, up to the first failing one, produce XML files . Find out and open them in the directory ${test.data.dir}, i.e. build\data includes="**/*mpleTest.class"/> ant -f mybuild.xml clean test-batch OMITTED: What if we have test suite AllTests.class in ${build.test.dir}? Will it run? Passing non-JUnit, or abstract, classes to Causes an ERROR [junit] No runnable methods and again the same with the change includes="**/*mpleTest.class"/> Do not forget to recover *mpleTest.class to *Test.class. Passing non-JUnit, or abstract, classes to results in an error. -Dbuild.debug=on All these running tests, including the last (failing) one produce XML files. TRY to confirm this on examples. Say, create also CopyofSimpleTest.java.

15 Notes on Terminology Unfortunately the terminology of various textbooks on Ant on of our lectures differs from that used in the in Ant’s console output: Our Test Cases are called Testsuites in Ant’s console output. Our Test Methods are called Tests in Ant’s console output. 4.7

16 Generating (HTML) test result reportsWith test results written to XML files, it is straightforward to generate HTML reports (by using XSLT). The task generates HTML reports : it aggregates all individual XML files (for each Test Case) generated from or into a single XML file named, by default, as TESTS-TestSuites.xml and then applies an XSL transformation to this file to get HTML report by using sub-task. 4.7

17 Generating (HTML) test result reportsGenerating HTML report consists of: placing the task immediately following the task (in the target test-batch described above): todir="${test.reports.dir}"/> XML files, in ${test.data.dir} are aggregated in the same directory into one XML file and transformed to HTML files in ${test.reports.dir} (or "noframes") = XSLT transformation done by sub-ask

18 Generating (HTML) test result reportsWe aggregate all generated "TEST-*.xml" files since that is the default naming convention used by the XML formatter of . Then HTML report is created according to the sub-element. But it will not work if haltonfailure="yes" and some test case fails.

19 Generating (HTML) test result reportsAdd the above task in test-batch target immediately after closing tag and before end tag . Put temporary haltonfailure="yes" RUN it: ant -f mybuild.xml clean test-batch Was task started working to create HTML report in build/test/report? Put haltonfailure="no" , try again and compare the results. How can you explain the difference? Was HTML report created now in build/test/report? But again, BUILD SUCCESFUL whereas some tests FAILED!? Everything looks good, except the last, not very natural point ant -f mybuild.xml clean test-batch OMITTED: -Dbuild.debug=on (Instead of -Dbuild.debug=on we could set this property in mybuild.xml)

20 But this does not allow to createGenerating all test reports and enforcing the build to fail in case of failures We know that haltonfailure="yes" forces build to fail and, actually, to halt if any of the tests fails. But this does not allow to create all TEST-*.xml files, the aggregated TESTS-TestSuites.xml file, and the HTML report. As a solution, turn off haltonfailure in order for the XML and HTML reports to be generated as above before the build halts, and additionally, enforce build failure, after generating XML and HTML reports, by setting a specified property test.failed upon a test failure or error: use the failureProperty and errorProperty attributes of the task, and conditional task. 4.7.1

21 ( part continued on the next slide)Generating all test reports and enforcing the build to fail in case of failures Create new target test haltonfailure="no" errorProperty="test.failed" failureProperty="test.failed"> includes="**/*Test.class"/> The most important target for Lab Test test.failed is a property which is set to true in case of error or failure As in test-batch above haltonfailure="no“ can be omitted at all since "no" is default This is only testing part ( part continued on the next slide)

22 Generating all test reports and enforcing the build to fail in case of failures todir="${test.reports.dir}"/> if="test.failed"/> End of creating part Conditional task enforcing build to fail (end of target) ant -f mybuild.xml clean test RUN C:\Antbook\ch04>ant -f mybuild.xml clean test Now we achieved our goal: build fails if some test case fails, but HTML report has already been created before build failed!

23 Generating (HTML) test result reportsOpen C:\Antbook\ch04\build\test\reports\index.html Test methods Test cases the default frames This is the generated main page, index.html, by . It summarizes the test statistics and hyperlinks to test case details.

24 Generating (HTML) test result reportsNavigating to a specific test case FilePersistenceServicesTest displays: A specific test case results: Test methods and details of corresponding assertion that failed are clearly shown. Clicking the Properties » shows all of Ant’s properties at the time the tests were run.

25 Generating (HTML) test result reportsSelf-study Clicking the Properties » above shows all of Ant’s properties at the time the tests were run. These can be handy for troubleshooting failures caused by environmental or configuration issues. NOTE. There are some issues with and : has no dependency (uptodate) checking; it always runs all test cases (even if they are not up to date). simply aggregates all XML files without any knowledge of whether the files it is using have any relation to the tests just run (They could be old.) Use task (considered later) to ensure tests only run if things have changed. Cleaning up the old test results before running tests gives you better reports. Fig. 4.6 ??Requirements of ???

26 Running a single test case from the command-lineSelf-study While a project can have many test cases, you may need to isolate a single test case to run when ironing out a particular issue. This can be accomplished using the if/unless attributes on and . Our task evolves again: 4.7.2

27 Running a single test case from the command-lineSelf-study errorProperty="test.failed" failureProperty="test.failed"> todir="${test.data.dir}" if="testcase"/> These minor additions to our “test” target make the required effect.

28 Running a single test case from the command-lineSelf-study By default, testcase property will not be defined and, therefore, the will be ignored, and will execute all of the test cases. In order to run a single test case, run Ant using a command line like >ant test -Dtestcase= C:\Antbook\ch04>ant -f mybuild.xml clean test -Dtestcase=org.example.antbook.junit.SimpleTest TRY it and compare with the previous run and html result. ant -f mybuild.xml clean test -Dtestcase=org.example.antbook.junit.SimpleTest OMITTED: -Dbuild.debug=on

29 About testing again Now, having Ant and JUnit tools, we can summarise how test-driven programming can be done: Writing and automated running test cases may actually improve the design of our production code. In particular, if you cannot write a test case for a class, you have a serious problem, as it means you have written untestable code. Hope is not lost if you are attempting to add testing to a large system on later stages . Do not attempt to incorporate test cases for the existing code in one big go. (that was built without unit tests in place)

30 About testing again Before adding new code, write teststo validate the current behaviour to describe (specify) the expected behaviour on new code to be added, and to verify that the new code does not break the current behaviour and demonstrates the correct new behaviour. When a bug is found, write a test case or a test method to identify it clearly, then fix the bug and watch the test passes. While some testing is better than no testing, a critical mass of tests needs to be in place to truly realize such XP benefits as fearless and confident refactoring. Keep at it and the tests will accumulate little-by-little allowing the project to realize these and other benefits.

31 Self-study Extensions to JUnit Partly discussed earlier.It is easy to build extensions on top of JUnit. There are many freely available extensions and companions for JUnit. This table shows a few: Name Description HttpUnit A test framework that could be embedded in JUnit tests to perform automated web site testing. JUnitPerf JUnit test decorators perform scalability and performance testing. Mock Objects Allows testing of code that accesses resources such as database connections and servlet containers without the need of the actual resources. Cactus In-container unit testing. Covered in detail in chapter 12 of Ant book. DBUnit Sets up databases in a known state for repeatable DB testing. Table 4.1 Because of its architecture, it is easy to build extensions on top of JUnit.

32 BRIEF SUMMARY to Ant+JUnitJUnit is Java’s de facto testing framework; it integrates tightly with Ant. Ant task runs tests cases, captures results in various formats (e.g. XML), can set a property if a test fails. Ant task with sub-task generates HTML test result reports (from XML), allows for customization of the reports generated via XSLT 4.10 Unit testing makes the world a better place because it gives us the knowledge of a change’s impact and the confidence to refactor without fear of breaking code unknowingly. Here are some key points to keep in mind: Information can be passed from Ant to test cases via .???

33 BRIEF SUMMARY to Ant+JUnitThere is a lot more on Ant and JUnit what we had no time to discuss. Read Ant book and other materials presented in the Web site of COMP220. 4.10 Unit testing makes the world a better place because it gives us the knowledge of a change’s impact and the confidence to refactor without fear of breaking code unknowingly. Here are some key points to keep in mind: Information can be passed from Ant to test cases via .???