Resources | Resources |



Observations that can be documented as issues during item testing

Note: The following is not intended to address all possible observations that are made during testing.

In addition to the criteria for failing an item, the tester also documents in the test plan observations about the item. These observations are referred to as issues. Issues are not considered when determining whether an item passes or fails testing (errors are evaluated to pass or fail an item). However, a commercial channel may choose not to accept an item with issues. Issues are reported if the item deviates from the following ideal behaviors:

  • Drawing screens—Each screen (e.g., splash, help, pop-up boxes, and text entry) repaints correctly and with the correct contents. Repaints occur correctly after dismissing a screen that overlays another one.
  • Screen contents—Screen contents are clearly displayed for easy user viewing. Directions and information can be read and understood. Error messages displayed by the item under adversarial conditions (e.g., loss of service) are not confusing.
  • Keypad input—Keypad functions correctly navigate through the item. Keypad functions are handled correctly on each screen (e.g., splash, help, pop-up boxes, and text entry).
  • Text entry—Keypad entry fields handle input text correctly. Keypad entry is handled correctly on each screen (e.g., splash, help, pop-up boxes, and text entry), and the item does not accept invalid input values.
  • Scroll bars—Scroll bars work correctly on each screen (e.g., splash, help, pop-up boxes, and text entry).

In addition, testers will always note the following:

  • Resource usage—Number and size of files created by the item in the item directory and shared directory.
  • Inconsistency of keypad input, text entry, scroll bars, and/or button designs.

If they are serious enough, deviations from the above behaviors may be classified as errors. For example, if a screen does not repaint correctly, making it difficult or impossible to use the item, the behavior is reported as an error that fails the item. If the repaint problem is noticeable but does not affect usability, the deviation is reported as an error; and the item receives a test result of Passed with Notes. If the repainting problem is barely noticeable, it is likely to be reported as an issue, and the item receives a Pass test result if there are no errors.