A beginner’s guide to reporting and tracking defects
Congratulations! You’ve landed a job at a software or Web development company. You might have heard about a bug tracking system during your first few days at the office, but that has nothing to do with you, right? Well, think again.
Are you in customer support? If so, you will be responsible for logging customer problems as they are phoned, written, or emailed in.
As a developer, you’ll be strictly adding new features or enhancing existing ones, right? Wrong—you’ll also be responsible for fixing any known bugs.
If you are a technical writer, your job doesn’t end with creating Help, Manual, or Web content. You’ll also be fixing any documentation errors.
If you’re a manager, you’ll need a way of assessing the viability of the product. You’ll also need to determine what work remains to be done and how soon a product can be posted or shipped.
And if you are a tester working in the SQA (Software Quality Assurance) department, your job will be to verify the integrity of the product, and log and manage the resolution of any problems that you or anyone else may find.
All of these people require the use of a bug tracking system to accomplish their tasks. If you’re one of those people and need to learn more about defect tracking, keep reading. This article will tell you what you need to know.
What Is a Bug Tracking System?
A bug tracking system is constructed as a database. The system may be based on an existing database such as Lotus Notes, or it may use a built-in proprietary database. The bug tracking system may be homegrown or it may be a commercial system. In any case, a bug is entered as a record into this database, where it can be assessed and tracked.
The user interface usually consists of navigable views and forms for editing existing bug reports or writing new ones. Each user is assigned a unique login ID and password. Whenever a user writes up a bug, adds comments, or changes a status field, a good tracking system will log the ID of the person, creating or amending the record. A great system will save such history as read-only data so a true history of a bug report can be maintained.
A good tracking system will give each member of the team easy access to bugs assigned to them. The design should make it easy for the team members to manage their specific bugs effectively with the data sorted in many different ways. A great system will give management a variety of views and reports reflecting the health of the product, so they can assess the product situation on a build-by-build basis.
Of course, some users of the system are given more rights than others. For example, most users aren’t allowed to delete bugs. If a bug is a duplicate or entered in error, it is simply marked as such instead of being deleted.
One of the most important features of a bug tracking system is keyword searching. For example, let’s say a user finds a crash bug while saving a file. The bug tracking system should enable the user to perform a keyword search on "crash and file." If a match isn’t found, the bug can be entered. If existing bugs are found, the user should examine them to see if the report covers his circumstance.
The Bug Lifecycle
Let’s examine a bug’s lifecycle:
-
Someone finds a bug and enters it into the system
-
The bug is assigned to a developer
-
The developer looks at the bug
When a bug is entered, the author usually assigns it to a product category such as "user interface" or "calc engine."
Writing or entering a bug is also known as opening a bug. Some systems may default a bug to "New" and wait for someone to open it. From my experience, doing so is a wasted step.
Depending on how the database is constructed, the bug may be assigned automatically to a developer and QA person according to the chosen category. It’s up to the developer to look for open bugs with her name on them. It is up to the development manager to look for open, yet unassigned bugs and assign them as soon as possible. Once a bug is assigned, most systems will send the developer an automatic email notification. Bugs may be reassigned if it is determined that the default person is not the best person to fix the problem. Sometimes, if a developer sees an unassigned bug in her area, she will take on the assignment and proceed to work on it. A QA person should be just as proactive if he sees unassigned bugs in his area as well.
Once a developer takes ownership of a bug, she assesses the situation and sets a resolution field. The bug’s state is pending after at least one person (usually the developer) comments on the bug or sets the resolution field. From this point on, the bug remains pending until someone either closes or verifies it.
A bug is verified when the original steps no longer reproduce the problem and someone in QA has reviewed the fix. But this doesn’t mean that the bug can be closed. The bug is usually kept in this state until QA has done a thorough check under all supported operating systems.
If verification fails, what happens to the bug is a gray area. Some systems may revert it back to "open," or set some other flag so it lands back at the feet of the developer or development manager.
It is up to the QA person assigned and/or the author of the bug to review the resolution and decide if they agree.
QA usually has the responsibility of closing a bug. This happens when either they are satisfied that the bug has been properly verified, or they agree that the bug is no longer (or never was) an issue.
Resolutions
A resolution can be one of the following:
Fix—The developer thinks she has a fix to the problem. A field indicating which build the bug should be fixed in will accompany this. The QA person assigned the bug should wait for that build to be released, then verify whether the fix actually works. If it doesn’t work, this isn’t a plot by the developer to fool anyone. When a developer makes a fix, it may be difficult for her to verify it until the build is released.
Defer—A bug is usually deferred when management or all parties agree that the issue isn’t drastic, and can be put off (deferred) until a later time.
Dupe (Duplicate)—This means that someone else has already reported the problem. When you first start on a project, you will see this in your own bugs more often than you would like. One way to avoid starting out with many dupes is by spending an afternoon reviewing the bugs that are currently open. Mastering your system’s keyword searching capabilities is another way of avoiding too many dupes.
A good system will have a field for the original bug number. The person marking the bug as a dupe should fill in the number of the original to make it easy for others to go back, review the original, and see if they agree. It’s quite possible that the person marking the dupe is assuming too much. If you think the bug is not a dupe, you should reset the resolution field and note as such.
Need More Information (NMI)—This means the developers don’t have enough information to understand or reproduce the problem. They should note in the bug what they are looking for.
Not a Bug (NAB)—The developers disagree that it’s a bug. Another way of saying this is Working as Designed (WAD).
Not Reproducible (NREP)—The bug could not be reproduced. This may be either because there is something strange about your setup (you have a driver or service pack that the developer is missing, or vice versa), or the person looking at the bug is trying it in a later build. The bug may have been inadvertently fixed since you encountered it in the last build.
However, if you can still reproduce it on your machine with the later build, you should disagree, reset the resolution, and talk to the developer.
No Plan to Fix (NPTF)—The bug has been recognized as a problem, but the developers don’t plan to do anything about it. You’ll see a lot of this as the product approaches its ship date and the pressure mounts. If you disagree with the resolution, you may need to bring it to your boss’s attention.
User Error—This is a polite way of saying that the problem isn’t with the product. User error is sometimes referred to as UBD (User Brain Damage).
Bug Components
A typical bug report contains the following components:
Bug Number—A bug report contains a unique ID, so it can be tracked. Whenever you tell anyone about a bug, the first thing you’ll be asked for is the bug’s number or ID. The number or ID allows the developer to enter the database and look it up.
Short Description—The short description may also be referred to as the "title" of the bug. This is one of the most important fields to get right. Filling it in with "Product doesn’t work" will probably incur the wrath of not only your peers, but also your boss. When sifting through a list of bugs, the short description needs to be clear, concise, and to the point. One example is "Crash on exit only after save." That will get their attention. You will notice that the description doesn’t need to be a complete sentence.
Product/Project Name—Versions of a product are usually given code names. For example, Lotus 1-2-3 version 3.0 was code-named "Godiva." Dragon NaturallySpeaking version 3.52 was code-named "Yukon."
There was a time when code names were used to keep outsiders guessing as to what secret project people were working on. But you will notice more often that code names are being used in press releases, along with the actual product and version number.
Product Version— The product version is the public designation for when the product ships (such as Dragon NaturallySpeaking 3.52). Version numbers usually include point-release identifiers (3.01 versus 3.02). It’s usually a good idea to keep the product version number as a separate field in the bug report. Some companies assign different project names to different versions. If the point release only contains one or two bug fixes, some companies won’t bother assigning different project names.
Build Number—Every time someone (usually a developer or someone designated as the build/release engineer) recompiles the product and incorporates the latest bug fixes, it is given a new build number. The build in which a bug was found should be noted in the bug report.
Fixed Build Number—When a developer fixes (or hopes she has fixed) a bug, she must note in the bug report which build the fix will be in. The fix can’t be verified as working until a copy with that build number or higher is released.
Steps to Reproduce—Before you write one line on how to reproduce a bug, you should attempt to narrow any extra steps that are not absolutely needed to reproduce it. (If you’ve ever used a debugger or watched a developer use a debugger, you’ll know why.) The developer might need to walk through every line of code as it executes for each step. For instance, clicking File/Save may seem trivial, but that could launch a very long sequence of events or large loops that the developer needs to step through or wait for.
Comments—Additional comments should be direct and to the point. Anything not directly related to the bug should be reserved for email.
Author—Most systems will fill this in automatically, based on the login ID.
Operating System—This is very important. If a developer can’t reproduce your bug, it could be because she’s using a different operating system or version.
Web Browser—If you are testing a Web site, the browser you’re using is as important as the operating system.
Category—Always try to define a category for a bug because this helps in assigning it. If someone thinks that you chose the wrong category, that person will most likely correct it.
Subcategory—Some categories may not have a subcategory. If you think a subcategory is appropriate, then you should select it. You’ll need to pick one if your system demands that a selection be made. As with the category, if you make an incorrect choice, someone else will probably correct it.
Developer Assigned—The developer field should contain a list of developers to choose from. Some systems will assign a developer by default, based on the category. Other systems may leave this field as "unassigned." If a bug is left unassigned, it’s usually the responsibility of a manager or the developers to assign the bug to the appropriate person.
You should be diligent about the bugs you enter and make sure they don’t stay unassigned for too long. If this is the case, find out what your organization’s protocol is for assigning unassigned bugs. It may simply be that the mechanics of automatic assignment are not in place, allowing you to assign the bug yourself, working either from a list or common knowledge. If the assignment is not appropriate, someone will correct it.
If you are a developer and a bug is erroneously assigned to you, you should assign it to someone else; assign it to your boss and let him figure it out; or revert it back to an unassigned state (if possible). Whenever you change the assignment of a bug, you should leave a simple note as to why you’ve changed it so the bug isn’t reassigned to you.
Developer Resolution—Development sets this field after they have looked at the bug.
QA Assigned—The QA field should contain a list of QA engineers. As with the developer field, some systems will assign this field automatically. If it isn’t assigned automatically, then a manager will assign the field or QA people can assign bugs in their area to themselves.
State—Open, Pending, Verified, and Closed are the most common bug states.
Priority—Some bug systems allow a priority field to be set. This may be a keyword list that includes Urgent, High, Medium, and Low, or a number whose value is defined by the department. This helps developers distinguish which bugs to fix first.
Severity—This field is used to define how severe a bug is. For example, a user being able to log in and see someone else’s credit card information may not be a crash bug, but it’s still a severe problem.
History—A good bug system will record all comments and field changes, as in a read-only history field that anyone can review, but not edit.
Attachments—These can include bitmaps, files, anything you want. When attaching anything to a bug, make sure it’s in *.zip format. Quite often, attachments stretch hardware limits and can slow down the system. Many organizations insist on storing attachments on public servers and just leaving a reference in the bug.
Miscellaneous/Custom—Various departments may have other fields added to bug reports for their own use.
Tips for the Beginner
You may be new to SQA, but you probably know that many people are rated on how many bugs they find. There is an implied time limit for how long you should spend chasing down one bug, so you shouldn’t let one bug consume your life. If you can’t nail it down, bite the bullet, write a report where you state as much as you can, and move on. If it’s a nasty crash bug, tell your boss and work out whether it requires more attention. If so, enlist the aid of others to track it down.
Because of this rating system, some people new to QA may get the urge to "stretch" a bug by making several bugs out of one. An example of this would be writing a bug where the user enters some text, noting the product crashes, then writing up the same bug, but entering different text. Don’t do this. You’ll just waste everyone’s time, and your extra bugs will probably be flagged as dupes.
Before writing up a bug, ask Development what’s fair and what isn’t. "Fair game" refers to areas where it is fair for QA to write bugs. If Development tells you that the File Import routine is still under construction and not guaranteed to work, this isn’t fair game. Writing up bugs against this area is another waste of time and can make people angry.
Another tip is to review the bug system as a whole. Examine the trends and try to get a feel for what areas are vulnerable and need more testing. If the bug count increases over time (builds), this is a bad thing. If the bug count decreases, it could mean that the developers have kidnapped the QA team, but it’s more likely to mean the product is becoming more stable (and approaching a date when it can ship/post).
One more thing to look out for is developers going wild by marking everything (including crash bugs) as NPTF (No Plan To Fix), NAB/WAD (Not A Bug/Working As Designed), or NREP (Not Reproducible).
Management’s Role
The first thing a manager should do is to make sure that all bugs are assigned. But since managers are usually stuck in meetings all day, team members need to be proactive by taking on the unassigned bugs themselves.
Managers should do everything they can to squeeze data out of the bug tracking system—especially graphs. Although I could dedicate a whole article to the types of reports and graphs that are needed, you basically need to chart how the product is doing over time. Time can best be mapped out by build number.
Final Thoughts
Nothing in this article is cast in stone. I’ve tried to provide an overview of a generic system, based on my experience of using and building bug tracking systems. As you learn your company’s system, you may find that items and procedures are a little different.
Regardless of the system you are using, mastering its intricacies can help you to better manage your workload, generate comprehensive reports, and track progress for yourself and your department.
For more information check out the StickyNotes from this author
• Further reading
Terminology
Bug—A bug is another term for describing a defect in a software program. If you start the product and it crashes, that’s a bug. If you find a typo in the Help system, that’s considered a bug too.
Bug Report—You may have also heard the initials "SPR" (Software Problem Report), which is the same as a bug report.
Bug Tracking—Some people refer to a bug tracking system as a "defect tracking system." It’s the same thing. I’ve even seen it called a "nonconformity reporting system."
Build—Every time Development recompiles the code, it is considered a new "build." Each build is assigned a number or an alphanumeric string indicating some sort of sequence. It is imperative that for each bug written, there is a note of which build the bug was found in, as well as which build it was fixed in. (Because Web site development is more fluid, assigning a build number may be difficult. Some companies may assign build numbers to different components of a Web site rather than to the whole site itself.)
Development—This can also include the documentation group. Both parties are responsible for developing something (software versus content), and they are usually lumped in the same category.
Product—When I refer to a product, it can be either a software application or a Web site.
QA, QE, SQA—These are all the same thing. QA stands for Quality Assurance. QE stands for Quality Engineering (usually used when a QA team wants to be given the same respect as the developers). SQA stands for Software Quality Assurance. SQA is often used on job boards to distinguish software quality professionals from quality control officers in medical and manufacturing industries.
Releasing a Build—Before a product is shipped to customers or posted to a public Web site, it goes through many builds that are released internally for testing. This means that development has successfully compiled the latest bug fixes and/or product updates and is releasing them to the rest of the team. They may have run a simple test to verify basic functionality, but that’s usually all that is guaranteed. It’s up to QA to verify whether the build is healthy enough for general testing.
Shipping—When I refer to shipping, I am talking about releasing a software product to the public. It can also refer to posting a Web site to publicly available servers.
Tester—This is either an SQA professional or anyone else drafted to help test a product.
And a few words about Regression—I’d like to point out that a "regression" is not a bug. If a developer tells you that they’ve "introduced a regression"—well, that’s because they don’t want to come out and say that they broke the product. Defining what regression really means is outside the scope of this article.
You do not have Permission to view this Post
