| <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
| |
| <html> |
| <head> |
| <title>Q-Evangelism</title> |
| <meta name="DESCRIPTION" content="Describes tools developers can use to increase the quality of NetBeans codebase."> |
| <link rel="stylesheet" type="text/css" href="../../netbeans.css"> |
| </head> |
| <body> |
| |
| <h1>Quality Evangelism</h1> |
| |
| <p><b>Motivation</b>: Increase the quality and reliability of our codebase - the NetBeans modules. |
| </p> |
| |
| <p>When developing - writing new code, refactoring or just bug fixing - people always make mistakes and produce |
| new bugs. This is an unavoidable fact and there is almost nothing we can do to prevent ourselves from making |
| mistakes when coding (perhaps writing less code). That's why we should carefully learn how to find our own |
| mistakes reliably and quickly. The sooner the bug is found and fixed the better, because once the bug is fixed |
| can no longer negatively influence further work. The best solution would be to find and fix all bugs before the |
| code containing them is committed into the version control system and the bugs can disrupt others who use the code. |
| <p> |
| |
| <p>Every developer is excited when designing and writing new features and every developer hates fixing and |
| code maintenance. In reality developers spend significantly more time on bug fixing than developing |
| new features, don't you think? Why? They haven't found their own mistakes before putting the code into the user's hands! Finding |
| bugs immediately after writing new code will give you more time for new features and for fun. Besides that, it will |
| make your users happy and your boss satisfied. Yes, so much good can come just from having no bugs. |
| </p> |
| |
| <h2>How to find bugs?</h2> |
| |
| <p>In NetBeans there are several instruments which we use to find bugs. They are focused on different |
| levels of testing, use different methodology and have different complexity, but in the end they all |
| test the code and are capable of detecting bugs. Hopefully the list isn't final and will soon grow. |
| </p> |
| |
| <h3>Milestones Process</h3> |
| |
| <p> |
| The NetBeans <a href="http://qa.netbeans.org">QE team</a> releases <a href="http://wiki.netbeans.org/MilestonesProcess">Milestones</a> to assure a basic level of quality during the release cycle. |
| The Milestone Process is well-established among the developer community. Milestones reports are published on the NetBeans site and announced on the <a href="https://netbeans.org/community/lists/top.html">nbdev@netbeans.org mailing list</a>. There are strict rules followed by all developers about when and how to fix Milestone stopper bugs. |
| </p> |
| |
| <h3>Commit Validation Suite</h3> |
| |
| <p>The Commit Validation Suite should help all developers to verify that the changes they are going to push |
| to the NetBeans Mercurial repository don't break any vital functionality provided by the NetBeans modules. This is the suite |
| of tests gathered among NetBeans modules which developers should run before pushing their changes to the Mercurial |
| repository. Remember, your bugs may disturb other developers around the globe. |
| </p> |
| |
| <p>The Commit Validation Suite can be extended by any module whose functionality the developers consider |
| vital and should be tested before every commit. You can read more about |
| <a href="https://netbeans.org/community/guidelines/commit.html">commit guidelines</a> here. |
| </p> |
| |
| <h3>Unit tests</h3> |
| |
| <p>The NetBeans developers and QA team invested a lot of time in implementing an easy-to-use framework allowing |
| to write and run unit tests. Nowadays I doubt there are any developers who have never heard of |
| unit testing or think that writing unit tests is a bad thing. |
| Unfortunately, the reality is that even those developers still do NOT write |
| any tests. Why? They haven't discovered the benefits. |
| </p> |
| |
| <blockquote> |
| <h4>Why write unit tests?</h4> |
| |
| <ul> |
| <li>Tests represent the most practical design spec possible - |
| Unit tests constitute design documentation that evolves naturally with a system. This is the Holy Grail of software |
| development, documentation that evolves naturally with a system. What better way to document a class than to provide |
| a coded set of use cases. That's what these unit tests are: a set of coded use cases that document what a class does, |
| given a controlled set of inputs. As such, this design document is always up-to-date because the unit tests always |
| have to pass. |
| </li> |
| <li>Tests give a developer confidence in the code - you can always check whether your change breaks something |
| or not by running tests. Well-written tests should catch all potential regressions. |
| </li> |
| <li>Tests are the lab - when testing your code this is usually done in laboratory environment which eliminates |
| dependencies and side-effects caused by other parts of code. You can test each functionality separately which will |
| allows you to quickly track the problem. You can't expect your code to work in the production environment if |
| it doesn't in the lab. |
| </li> |
| <li>Tests speed up refactoring - by coding a set of use cases into your tests you can easily verify that you |
| are not compromising any functionality or causing incompatibilities when refactoring your code. Well written tests |
| should catch all potential incompatibilities. |
| </li> |
| <li>Tests help to fight memory leaks - it's usually hard to find memory leaks in a program, and it's harder when |
| the program is big and complex. Running tools like OptimizeIt doesn't necessarily help in all cases and it's usually |
| slow. It's better to write tests for isolated pieces of code (subsystems) which check whether your objects |
| are really getting garbage collected when they aren't used anymore. |
| </li> |
| <li>Tests will make your boss more happy - by writing tests your code will be more stable attracting more users, |
| making the product more successful. That's what all bosses like! |
| </li> |
| </ul> |
| |
| <h4>How many tests do I need to write?</h4> |
| |
| <p>All of them, and it still will not be enough! |
| Seriously, you should write as many tests as possible, ideally the tests |
| should cover all the code in your module. Practically this is hard to achieve and software projects |
| usually define some lower level of code coverage as sufficient. However, the code coverage by unit tests |
| should be at least 70% for production quality code. This is a high number! |
| </p> |
| |
| <p>The problem with code coverage by tests is that even 100% coverage doesn't mean that there are |
| no bugs in the tested code. This is because your code behaves according to the state of memory and thus |
| you shouldn't only exercise each line of your code but exercise it for each possible state of memory. |
| Obviously threading issues bring even more complexity to the problem. |
| </p> |
| |
| <h4>When to write Unit tests?</h4> |
| |
| <p>In NetBeans, currently we have a lot of code which isn't covered by tests. Trying to write tests for all |
| this code sounds like a never-ending task and in fact it's not really practical to stop all work and write |
| tests for all the code now! Instead we should establish few rules about how to continuously improve the situation and |
| increase the number of code lines covered by tests. And here it goes: |
| </p> |
| |
| <ul> |
| <li>All newly written code should be covered by unit tests</li> |
| <li>When fixing a bug, a test should be written to simulate the bug and then verify that the fix really solved |
| the problem.</li> |
| <li>Whenever you feel tired by your current work try to write some tests for existing code which isn't tested |
| enough yet (there is a lot of untested code overall in the NetBeans sources).</li> |
| </ul> |
| </blockquote> |
| |
| <h2>Related Links</h2> |
| |
| <ul> |
| <li><a href="../../community/guidelines/index.html">NetBeans Community Guidelines</a></li> |
| <li><a href="../../community/guidelines/commit.html">Commit Rules: Pushing To The NetBeans Mercurial Repository</a></li> |
| <li><a href="../../community">NetBeans Community Home</a></li> |
| </ul> |
| |
| </body> |
| </html> |