Solved

Software Testing

Posted on 2004-04-12
10
342 Views
Last Modified: 2010-04-21
Hello Gurus:

I am new to software testing and I want to perform White box and Black box testing on a software tool called "CCount" . This testing is in Unix and will involve commands like mccabe, halstead,gcov

Can anyone please tell me what is the process? Please tell me as u r telling a novice. I will greatly appreciate it .
0
Comment
Question by:smartrider
  • 3
  • 2
  • 2
  • +2
10 Comments
 
LVL 11

Expert Comment

by:avizit
Comment Utility
Don't know about the others but for 'gcov' read

http://gcc.gnu.org/onlinedocs/gcc-3.0/gcc_8.html


/abhijit/
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Do you want to use profiler to look for inefficiencies in code or find bugs in your programs using regression ??? Or you look for automated static source code analysis tools ???
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Found your CCOUNT - it is written in C.
Look for splint package and gcc -Wall
0
 

Author Comment

by:smartrider
Comment Utility
I want to Black box Testing and White Box Testing. After that I have to do regression test as well.

I have the ccount package - it has a lot of ".c" and other functions in them

How can I test them ?
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 6

Expert Comment

by:bira
Comment Utility
****************************************************************************  

            R E A D M E  FILE  of  'C C O U N T'  PROGRAM

****************************************************************************  

==================================================================
0. Synopsis
============================================================================

      1. Description of 'ccount' program
      2. Input capabilities of 'ccount' program
      3. Behaviour in case of syntax error
      4. Output format of 'ccount' program
      5. How to apply 'ccount' to a set of source files



============================================================================
1. Description of 'ccount' program
============================================================================

usage: ccount [-v] <basename.c>

ccount supplies results of the structure of a C source code, e.g. length
of functions and control structures, block nesting depths, number of operators
in conditions and expressions, number of comment bytes etc.
ccount reads 'basename.c' for the C source code and 'basename.tpp' for a list
of type names and other words and writes the results to 'basename.out'.
The format of 'basename.out' is described in the fourth section.

'basename.tpp' need not exist, but it gives you the possibility to resolve
several kinds of parsing problems. It consists of (one entry per line)
     - type names (with no prefix)
     - preprocessor conditional directive (line beginning with '#')
     - name of macro declaration (line beginning with '&')
     - ignored identifier (line beginning with '!')

For example:
     #ifdef DUMMY
     display
     !PRIVATE
     &$DESCRIPTION
     HOSTINT
     &va_dcl

To go into details about 'basename.tpp', see the next section.

The -v option:

    This option can be useful when there is a syntax error and the
    reason is not obvious. You restart 'ccount' with this option and
    'ccount' will allow you to follow the lexical analysis. Each token
    and its token number will appear on the screen. Perhaps that happens
    too quickly for your eyes, but when it stops because of a syntax
    error, you have what you want to see: the last tokens before the
    error. You can now compare them with the assignments of tokens and
    their numbers in 'y.tab.h'.
    In addition, the numbers of five selected lexical rules concerning the  
    identifier context will be indicated (for reasons of internal action
    the same rule will probably be indicated more than once). These rules
    determine whether the current identifier is a possible type name or not,
    except rule 5 which recognizes constructions to declare function
    prototypes in ANSI as well as Kernighan/Ritchie style like
      extern void error A((char* msg, int level));
    For more details look at 'ccount.l', the LEX specification file for
    the scanner.
    Note: with the -v option 'ccount' takes longer.



============================================================================
2. Input capabilities of 'ccount' program
============================================================================

There are some restrictions on the kinds of source code files that can
successfully be parsed by 'ccount'.
The main reason is that the preprocessor is not run, because we must assume
that included header files are not available.
Since comments would be removed by the preprocessor we besides could not
assign comments to functions even if we count comments seperately before
the whole analysis.
Not running the preprocessor has still another effect:
Alternative implementations of complete function fragments that are enclosed
in #ifdef something ... #else ... #endif are counted completely, i.e., all
alternatives are treated as part of the function. This is in the spirit of
the counting purpose and thus intended. (It is a feature, not a bug)
A similar statement is true for the non-expansion of macros.

On the other side we have some problems caused by that:


2.1  Recognizing unknown type names
-------------------------------------

     i.e. type names that are declared in a header file or by #define :
   
      Consider the following:
      
          BOOLEAN found;
          display *mywindow;

      If there is no comment within each declaration, 'ccount' recognizes
      'BOOLEAN' and 'display' as type names by the immediate context.
      'ccount' assumes an identifier to be a type name whenever this
      is appropriate and possible and retains it for the rest of the
        input file.

      Because of this, declarations and uses of objects that have the same
      name as a type cannot be parsed, no matter whether these objects are
      labels, struct fields, or variables.
      Note as well that multiplications in declaration parts can cause
      syntax errors, e.g.

            int Stack[MaxSize*FieldLength];

      But such problems are rare.

      Special cases:

      a. Some casts to non-pointer-typedef-types X can only be parsed
         successfully if a declaration of an X variable appears textually
         before the cast, or if the type name occurs in 'basename.tpp'.

      b. Same as with non-pointer-typedef-types as parameters in function
         declarations like

            extern void error (string);


2.2  Handling of #if-preprocessor directives in the following cases:
---------------------------------------------------------------------

      a. Comments that are not encapsulated in /* ... */ brackets but
           instead in preprocessor constructions such as #if 0 ... #endif or
           #ifdef somethingweird ... #endif or the like

        b. Construction variants that cause syntax conflicts, e.g.
           
           #if abc
           while (x < y) {
           #else
           while (x < y-1 && b != a) {
           #endif
             do_it();
           }

      If 'ccount' finds the preprocessor conditional directive (e.g. #if 0,
      #ifdef dummy) in 'basename.tpp', all following this directive in the
        source file up to the first '#else', '#elif' or '#endif' regarding
      this '#if...' is considered as preprocessor (PP) comment and thus
      ignored during the parsing. That is, in the example only

           while (x < y-1 && b != a) {
             do_it();
             }

      will be parsed.

      If the same '#if' occurs more than once in the source file, it
        it is not always necessary and appropriate to treat each correspoding
      section as a PPcomment. Thus, each occurence will be indicated
      together with the length (in line) of PPcomment to help you estimating
      whether you should modify the preprocessor condition in the
      source file and in 'basename.tpp' in order to better control the
      handling of PPcomments.
      Note that there can be real excesses of interlocking '#if's and
      '#ifdef's, so that you will probably lose track of what's going on.
      Then the best is you give up.


2.3  Macros that break normal C syntax
----------------------------------------

      These macros will cause parser errors. This problem is obvious
      for macros that for example rename literal elements of C, such as

        #define IF if(
        #define THEN ){
        #define END }
        #define ELSE else{

      or the like. But also

        #define RETURN_VALUE(x)  {...;}
          ...
        a = b + c;
        RETURN_VALUE(a)
        c++;

      See yourself if it is worth adding missing semicolons.

      The following exceptions can be handled by 'ccount':

      a. If it seems that a source file can only successfully be parsed
         by ignoring an identifier, you can force 'ccount' to do it so
         by writing this identifier with the prefix '!' in 'basename.tpp'.

         For example:

           PRIVATE char c;

         will be accepted if '!PRIVATE' occurs in 'demo.tpp'.
         Note that 'PRIVATE' is counted as normal identifier and yet it
         is still ignored for the whole input file.

      b. 'ccount' accepts macros in declaration parts if the macro names
         occour each with prefix '&' in 'basename.tpp' and if the macros
         have one of the following five forms:

           macro_name            (e.g. va_dcl)
             macro_name()
             macro_name();
             macro_name(list)
           macro_name(list);

         where 'list' consists of expressions and/or type names.

      c. Finally 'ccount' accepts constructions to declare function
         prototypes in ANSI as well as Kernighan/Ritchie style like

           extern void error A((char* msg, int level));



============================================================================
3. Behaviour in case of syntax error
============================================================================

For many source files 'ccount' will not work upon first trial because of
syntax errors. In these cases, 'ccount' halts with an error message that
looks for example like the following:


"demo.c", line 70: parser error: syntax error at or near 'calloc'
-------------------------------------------------------------------------
           /* reserve_memory_and_complain_if_necessary  (Level 1) */

           b      = malloc (b_size);
         
>>         r      = (REFS)calloc ((maxref+1), sizeof (REF_INFO));
           l      = (LINES)malloc ((maxline+1) * sizeof (LINE_INFO));

           s_root = malloc (s_size);
         
-------------------------------------------------------------------------
Please enter type name OR
preprocessor conditional directive beginning with '#' OR
name of macro declaration beginning with '&' OR
ignored identifier beginning with '!' OR
just RETURN for giving up!
'@' at the end will repeat this request:


that is, 'ccount' gives you a message and the according source line with
four additional lines of source context before and after it. It then prompts
you for the entry in 'basename.tpp'. If you enter just a RETURN, the program
terminates. If you enter something else, it is appended to 'basename.tpp'
and then the program quits. You have to restart the run for this file,
perhaps with the -v option, until it terminates successfully.
Sometimes you have nevertheless to load the input file in an editor for
getting more information and for modifying a little the source file if it's
tolerable as well as worthwhile.
For more support consult the previous section.



============================================================================
4. Output format of 'ccount' program
============================================================================

ccount operates on one source file, say 'basename.c', at a time and
produces one output file 'basename.out'.
This output file contains exactly one line per object in the source file.
The object types considered by ccount are

  - module   (i.e. the whole source file)
  - function (where its nesting depth is defined as 1)
  - block    (statement sequence between { } WITHIN a function body, nesting
              depth > 1, except innermost blocks)
  - innermost block (block with no blocks inside it)
  - 'if' statement
  - 'while' or 'do' statement
  - 'for' statement
  - 'switch' statement
  - expression statement

An operator is counted as 1, including ?: , () , [].

The count of block lines begins with the first '{' and ends with the
according '}'. So therefore, in the following example

      if (i == 3) {
            do_it();
            i++;
           }

the block length is equal to the length of the 'if'-statement, that is,
four lines.

We will now describe the output format for each of these types
individually. The common property of all these formats is that they
consist of fields that are separated by one or several spaces or Tabs
and that the contents of each field are a contiguous string of
non-blank non-tab characters (most, in fact, are decimal integer numbers).


Module:
  1. 'M'
  2. Length (in bytes)
  3. Length (in lines)
  4. Comments (in bytes)
  5. Preprocessor comments (in bytes)
     (see 2.2)
  6. Number of identifier uses
  7. Total accumulated length of identifier uses (in bytes)
  8. Number of keyword or symbol uses
  9. Total accumulated length of keyword/symbol uses (in bytes)
 10. Number of preprocessor directives
 11. Total accumulated length of preprocessor directives (in bytes)
 12. Total number of ';'
 13. Number of functions
 14. Average innermost block length (in semicolons)
     (blockless functions are also considered here, see Function (13.))
 15. Standard deviation of the above
 16. Maximum innermost block length (in semicolons)
 17. Average average block nesting depth per function
 18. Standard deviation of the above
 19. Maximum average block nesting depth per function


Function:
  1. 'F'
  2. Length (in bytes)
  3. Length (in lines)
  4. Comments (in bytes)
  5. Number of ';'
  6. Number of 'if' statements
  7. Number of 'while' or 'do' statements
  8. Number of 'for' statements
  9. Number of 'switch' statements
 10. Number of 'goto' statements
 11. Number of labels
 12. Number of all blocks
 13. Average innermost block length (in semicolons)
     (is set to the value of (5.) if there are no blocks)
 14. Standard deviation of the above
 15. Maximum innermost block length (in semicolons)
 16. Average block nesting depth
 17. Standard deviation of the above
 18. Maximum block nesting depth


Block (which is not an innermost block):
  1. 'B'
  2. Length (in lines)
  3. Number of ';'
  4. Nesting depth
  5. Total number of enclosed blocks
  6. Maximum absolute nesting depth of enclosed blocks


Innermost block:
  1. 'I'
  2. Length (in lines)
  3. Number of ';'
  4. Nesting depth


'if' statement:
  1. 'i'
  2. Length (in lines) total
  3. Number of operators in 'if' condition
  4. Length (in lines) of 'then' part
  5. Number of ';' in 'then' part
  6. Length (in lines) of 'else' part
  7. Number of ';' in 'else' part
  8. Nesting depth


'while' or 'do' statement:
  1. 'w'
  2. Length (in lines) total
  3. Number of ';'
  4. Nesting depth
  5. Number of operators in 'while' condition
  6. Number of 'break' or 'continue' statements for this 'do' or 'while'


'for' statement:
  1. 'f'
  2. Length (in lines) total
  3. Number of ';' (including those in head)
  4. Nesting depth
  5. Number of operators in second expression of head
  6. Number of break statements for this 'for'


'switch' statement:
  1. 's'
  2. Length (in lines) total
  3. Number of ';'
  4. Nesting depth
  5. Number of operators in head expression
  6. Number of case labels
  7. Number of break statements for this 'switch'


expression statements (i.e. expression followed by ';'):
  1. 'e'
  2. Number of operators
  3. Nesting depth



============================================================================
5. How to apply 'ccount' to a set of source files
============================================================================

Probably you want to get results for many source files.
Remember that 'ccount' can treat only one input file.
Therefore in addition there is 'ccounter', a shell script that
analyzes a set of source files using 'ccount'.
But 'ccounter' can do still more:

     a. You can set a step size in order to select only a certain fraction
      of source files, for example: if the step size is 2, 'ccounter' will
      take only every second one of the given source files.

     b. If there is a syntax error you have two additional options after the
      request:
      - If you type '%', the current source file will be parsed again.
        (Meanwhile you have edited the source file in another window.)
      - If you type '*', the current source file will be parsed again with
        the -v option.
      Note that these two options are not indicated, yet they still are
      available.

     c. No matter what your entry is after a parser error, if it is more than
      a RETURN, the same source file will be parsed again automatically,
      otherwise 'ccounter' proceeds to the next input file.

As you see, 'ccounter' is useful even if you consider just one source file.

Note that 'ccount' must be in the current working directory or in your
$PATH before you begin.


How to work with `ccount':
--------------------------

    i.  Prepare a directory tree containing the sources.
      Use whatever organization you would like for the tree.
      The best way is to you place all source files at the same level,
      so that you can catch them with a single wildcard expression, e.g.
      */*/*.c  for sources on the third level of subdirectories.

   ii.  Type for example

          ccounter directory1/*/*.c directory2/*/*.c

      The number of source files that 'ccounter' has found, is indicated,
      e.g. 48. Now 'ccounter' prompts you for the step size, e.g. 5.
      After that the selected source files are parsed and analysed one
      after the other. The current file name is indicated together with
      the current sequence number and the number of all selected files,
      for example

          1/9: directory1/project1/demo1.c
          Success. Results in 'directory1/project1/demo1.out'.

          2/9: directory1/project1/demo2.c
            Success. Results in 'directory1/project1/demo2.out'.

          ...

      If a parser error occurs, follow the explanations above to resolve
      the problem.
      Finally 'ccounter' gives you the message

          8 file(s) successfully parsed.

      and terminates.

  iii.  Now you can collect the results and generate summary statistics.
0
 
LVL 6

Expert Comment

by:bira
Comment Utility
OK
0
 
LVL 44

Expert Comment

by:Karl Heinz Kremer
Comment Utility
mccabe and halstead report source code complexity numbers. The more complex a source file is the more likely it is that you have bugs in it. You can therefore use these complexity values to identify which files require whitebox testing and which you can limit to blackbox testing. I usually don't like whitebox testing, because it's very time consuming. IMHO it's much more useful to perform code reviews during the implementation phase and have the developers do their own kind-of-whitebox testing.
My approach to good testing is graybox testing: You need to know enough about the implementation so that you can direct your blackbox testing towards the areas in the software you expect to be more error prone.  
0
 
LVL 61

Expert Comment

by:gheist
Comment Utility
Neither tool will prove your application is secure, they can simply do static analysis to find if somewhere something looks messy. Only people can prove application secure.

Black box testing (from security side) would be guessing what application does, maybe knowing all I and O channels that belong to application
White box on the other hand is same, but knowing full specifications.
In either case you do not have access to source.

Since you have source, you can analyze it with all tools provided (add both of mine to your toolset, they work too)
Splint shows signs of quick programming and suggests you the fix (not mystical number, but actual line to fix)

On the other hand you can avoid source code examination and push random input and output to your application until it crashes, and then bring up debuger, and go back to source code.

Read 1st line again.
0
 
LVL 44

Accepted Solution

by:
Karl Heinz Kremer earned 500 total points
Comment Utility
Gheist, white box testing is actually more than what you describe: To do a "real" white box test, you do have access to the source, and you look at the different paths the program can take through the code (this is where gcov comes in handy, because it allows you to measure the coverage you achieve. Black box testing is what you describe as white box testing: You base your test cases on the specification of the program (you are basically testing that anything that is user accessible - the functionality according to the specification or the requirements document - is working correctly). What you describe as black box testing is just random, unplanned and inefficient testing.

Split also just does a static check of the source code, which means that you are still lacking the dynamic component. If you want to use tools to analyze the program for potential errors, you need to add something that also checks for runtime problems (e.g. memory errors). But, in my opinion (as a developer and former QA engineer) it's not the responsibility of the QA department to run these tests, the developer should have used these tools during the development cycle to make sure that what goes to QA is good enough to be tested.
0

Featured Post

Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

Join & Write a Comment

My previous tech tip, Installing the Solaris OS From the Flash Archive On a Tape (http://www.experts-exchange.com/articles/OS/Unix/Solaris/Installing-the-Solaris-OS-From-the-Flash-Archive-on-a-Tape.html), discussed installing the Solaris Operating S…
Why Shell Scripting? Shell scripting is a powerful method of accessing UNIX systems and it is very flexible. Shell scripts are required when we want to execute a sequence of commands in Unix flavored operating systems. “Shell” is the command line i…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now