Resources | developer.brewmp.com Resources | developer.brewmp.com

Developer

resources

Test Manager Help

Test Manager is a command-line tool for loading and running tests on a target. Test Manager generates and stores test results in a database file generated for each workspace created in Results Manager. Test results are automatically refreshed in the Results Manager window as you are running tests.

Before using Test Manager, confirm that Strawberry Perl 5.10 is installed, not Active State Perl.

Test Manager is an application that runs the Brew MP Validation Kit (BVK) tests, a collection of API tests for Brew Level 1 (BL1) and Brew Level 2 (BL2). API tests are a type of test module examining a specific platform interface feature. An API test module/case examines the functionality of a class, interface, or interface feature as documented in the API Reference. Test Manager also runs Platform Validation Suite (PVS) tests, a collection of tests used to validate APIs that are not BL1 or BL2.

The following are the different types of tests:

  • Automated: Tests do not require user interaction.
  • Interactive (only in PVS): Tests require user interaction (e.g., determine success or failure and enter the result, perform some action needed by the test, press a key, change a setting, or reset the device.
  • Auto-interactive:
    • User-invoked (only in BVK): Tests require you to set up and run the tests. The test module then automatically verifies your interaction and fails or passes the test.
    • Applet-invoked (only in BVK): Test module performs setup and runs the tests. After the automated portion finishes, a log for that test is created in the workspace log folder and the instructions for performing the interactive part of the test display in the command window. The test then requires you to manually verify and enter the test result. For example:
       Log fs:/usermods/addrbook_addaitest/addrbook_addaitest.log copied to 
    C:/ProgramData/Qualcomm/Targets/FFA 6290 (1.0.2.488)/Workspaces/ws2/Logs
    /addrbook_addaitest-2012-02-03-13-21-19.log
    
       Removing test AddrBook_AddAITest
    
       Go to the device and using the native UI, navigate to the contacts database. 
    Look for the newly added record with name "Joe Qualcomm." If you find the new 
    record, then the test has passed; otherwise the test has failed.
    
    You are then prompted to enter the test result in the command window:
    Please enter the test result.
    [Pass, Fail, Unsupported]
    
    If you entered Fail, the result displays in the log as follows:
    Result: Fail
    :

Prerequisites

Install the following before launching Test Manager (testmanager.exe):

  • Latest Brew MP SDK and Toolset
  • Test Manager can be used to run PVS Test Modules and BVK Test Modules, but only executes PVS modules from Platform Pro versions 1.0.2.669, 1.0.3.894.1, and 1.0.4.455 and higher.
  • Launch Results Manager and create a workspace. (See the Tools Reference or Results Manager Help for more information.)

Running tests

  1. Launch a target or connect a device with Developer Mode enabled. See the Connect Technology Guide and/or Target Manager Help for more information on connecting a target and enabling Developer Mode.
  2. After creating a workspace in Results Manager, open a command line window and change directories to %brewmp_toolset%\bin.

    Note: If you create the workspace while in Administrator mode, you must also run the Test Manager command line window in Administrator mode.

  3. Run a module from the BVK and/or PVS by entering:

    testmanager -run=
    

    To run all tests in the BVK and PVS, enter:

    testmanager -run
    

  4. For more information on Test Manager command line usage, enter the following in the command prompt or see the Test Manager parameters table in this section:

    testmanager -help 
    

Test Manager parameters

Test Manager parameters are specified in the following format:

testmanager  ... 

Parameter Description
-listTargets or -lt Lists the device target directories that are available on the host. These are the connected devices that can be executed against.
-target=

Specify the target name that comes from the list of targets provided by running Test Manager with the -listTargets parameter. The target name does not have to be complete; a segment is acceptable.

Tip: You can change the physical target name to a more user-friendly name. See Device Settings in Target Manager Help.)

For example, specifying simulator connects to the first simulator found or providing 234 connects to the first device with 234 in its name (i.e. target name, 300234034098988). If no target name is specified, Test Manager uses the first target it encounters.

-listbvks Lists the BVKs that are available.
-listplatforms or -lp Lists the platform pro kits that are available.
-workspace= Required for any operations related to tests. The Workspace contains the BVK and PVS test kits.

-listTests -workspace=

Lists available tests that can be filtered by family and subfamily.

-listFrameworks or -lf

Lists the frameworks available to the BVK specified in the database. Use results as a parameter to -load and -unload.

-load=

-load=

Loads the specified test framework or module to the device and can be chained; e.g., -load=ctf -load=ctfapplet:

Test Manager will try to load a framework first if the string does not contain wild cards. If the framework is not found, Test Manager assumes you are trying to load a test module.

Load can be chained, i.e., -load=ctf -load=oat -load=addrbook.

  • When using Test Manager wild cards, placement of an asterisk in the test name affects test module selection.
    • The following example runs tests with names that start with addrbook:
      testmanager -load=addrbook*
    • The following example runs tests with names that end with addrbook:
      testmanager -load=*addrbook
    • The following example runs tests with names that contain the word addrbook:
      testmanager -load=*addrbook*

-run= -family= -subfamily=

-kit=> -runoption= -skip=

Runs the test(s). The following are the parameters:

  • -family: Runs all BVK or PVS tests in the specified family.
  • -subfamily: Runs all BVK or PVS tests in the specified subfamily of the family.
  • -kit: Runs either the BVK or PVS tests. Valid options are bvk or pvs.
  • -dpk: Provides the path to a device pack, typically used by PVS tests. The input is a .dso file.
  • -runoption: Runs all automated (a), interactive (i), or automated-interactive (ai) tests. Valid options are a, i, or ai.
  • -skip or -sk: Skips specified tests or test modules that may be problematic. Skips can contain wildcard characters.

  • The following example runs the AddrBookTest and then DBCFactoryTest:
testmanager -run=AddrBookTest -run=DBCFactoryTest

  • When using Test Manager wild cards, placement of an asterisk in the test name affects test module selection.
    • The following example runs tests with names that start with database:
      testmanager -run=database*
    • The following example runs tests with names that end with database:
      testmanager -run=*database
    • The following example runs tests with names that contain the word database:
      testmanager -run=*database*

Note: If this parameter is not specified, Test Manager automatically loads the framework to which the tests belong, run the tests, and then unloads the frameworks. This parameter is useful if many individual tests belonging to the same framework are run, becauseTest Manager loads it once, runs the tests, and does not unload until a Test Manager command with the -unload parameter is executed.

  • The following example runs the tests AddrBookTest, DBCFactoryTest, and all of the tests in the Databases family:
    testmanager -run=AddrBookTest 
    -run=DBCFactoryTest-family=Databases  

  • The following example runs all the tests in the Databases family:
    testmanager -run 
    -family=Databases 

  • The following example runs all the tests in the Databases family, Contacts subfamily:
    testmanager -run 
    -family=Databases -subfamily=contacts 
  • The following example runs all the tests in the Databases family, Contacts subfamily, that are contained in the BVK:
    testmanager -run 
    -family=Databases -subfamily=contacts 
    -kit=bvk 
  • The following example runs all the tests in the Databases family, Contacts subfamily, that are contained in the PVS:
    testmanager -run 
    -family=Databases -subfamily=contacts 
    -kit=pvs 
  • The following example runs all the tests in the default BVK and the default platform:
    testmanager -run
  • The following example runs the test module AddrBookTest and only the test cases specified (this is only available on OAT and UITF tests and executing this command on a ctfapplet or ctf test results in all tests running):
    testmanager -run=AddrBookTest 
    -testcase=AddrBookTest_RemoveAllRecs 
    -testcase=AddrBookTest_EnumRecordsOnCat

The default location for the -dpk parameter is %brewmp_testsset%\testframeworks\oat\inc.

When running Test Manager with -testcases as a parameter following the -run option, Target Manager executes only the specified tests.

When running Test Manager as follows, Test Manger executes the test module AddrBookTest and only the specified test cases. Using this parameter on a ctfapplet or ctf test runs all the tests.

Note: This is only available on OAT and UITF tests.

 testmanager -run=AddrBookTest 
-testcase=AddrBookTest_RemoveAllRecs 
-testcase=AddrBookTest_EnumRecordsOnCat

-continue or -cont

Reexecutes the last run command and runs the test directly after the last test that did not finish running.

This command is useful when a test hangs. Restarting the device, or pressing CTRL-C, and restarting Test Manager picks up where the last run left off. This command cannot be run with any other command line parameters.

-bfs=

Runs test modules found in an XML BFS file. This file contains a collection of test modules for certain features.

-unload=

Used to unload the named test framework from the device, and is mandatory. Unload can be chained; i.e.:

-unload=ctf -unload=ctfapplet

-script=

Place command line parameters in a file and executed. The parameters look like the command line equivalents separated by end of lines. You can specify more than one script file. They are executed in the order that they are specified on the command line.

Note: You can specify a script file within a script file, executing it recursively, but you will encounter an error if Test Manager tries to run the same script file twice.

The following is an example script file:

-load=ctf

-run -family=databases

-unload=ctf

-logDir=

Use in conjunction with -process. Copies any logs found in the specified logDir path into the workspace log directory: C:\ProgramData\Qualcomm\Targets\\Workspaces\\Logs. A path must be specified for this argument.

If you do not use this parameter, the logs are copied to these default locations:

  • Windows 7: C:\ProgramData\Qualcomm\Targets\\Logs
  • Windows XP: C:\Documents and Settings\All Users\Application Data/Qualcomm\Targets\\Logs\

-nologdelete=, -nld=, -nologdelete, -nld

Sets whether the device test directory and logs should be deleted. -nld is just an alias for -nologdelete. The -nologdelete and -nld parameters without the Boolean values will set it to On.

-process

Reprocesses OAT logs and updates the workspace. If the -logDir argument is specified, the logs found in that path will be copied to the workspace before processing. Uses a target and a workspace.

-getlogs Downloads logs from the connected device and an associated workspace. -clean Removes all test frameworks, applications, and supporting modules and data from a target device and associated workspace. -verbose or -v Generates verbose messages.