Strtok function

Posted on 2003-11-19
Last Modified: 2012-06-27
If anyone writes this before Saturday and it passes my main.c function test (can be requested) it’s worth 500 points.

The motivation behind this program is to implement the functionality of the tokenizer that is a part of all compilers, interpreters, parsers, and scripting languages. For example, the C parser identifies the following comma-separated list of tokens (which are the smallest elements of a C program that are meaningful to the compiler) from the expression a=i+++j: a, =, i, ++, +, j. For our program, we will use the tokenizer to interpret the contents of a null-terminated string as words by filtering out the non-alphabetical characters. The following is a description of the function that you must implement:
Write a function StringToken() with the following prototype:
char* StringToken (char *str, const char *set);
StringToken mimics the behavior of function strtok declared in the C standard library file string.h. The explanation for StringToken is a function that searches str for tokens delimited by characters from set. A sequence of calls of StringToken(str, set) splits str into tokens, each delimited by a character from set. The first call in a sequence has a non-null str. It finds the first token in str consisting of characters not in set; it terminates that by overwriting the next character of str with ‘\0’ and returns a pointer to the token. Each subsequent call, indicates by a null value of str, returns the next such token, searching from just past the end of the previous one. StringToken returns null when no further token is found. The string set may be different on each call.
Sample Program:
Here is a sample program that illustrates the use of StringToken to determine the number of words in a null-terminated string by filtering out certain non-alphabetical characters:
#include <stdio.h>
#define LINE_LEN            1024
char * StringToken(char *str, const char *set)
            /* … */
int main(void)
      Note the use of the escape sequence \" to represent double quotation marks within
the string. An escape sequence begins with the backslash '\' and is used to represent certain nonprintable characters ( for example: '\0', '\a' (bell)),
the single and double quotation marks and the backslash characters ('\'', '\"', '\\').
            char line[LINE_LEN] = “\”My goodness,\” she said, \”Is that right?\””;
            /* Set of non-alphabetic characters */
char set[] = “ .,?!’\”\t\n”;
            char *word = StringToken(line, set); /* find first word */
            while (word)
                  printf("<%s>\n", word);
                  word = StringToken(0, set); /* find next word */
            return 0;
The results of the above program are the words (delimited by <>):
The function StringToken requires a variable defined with the storage-class specifier static. Block-scope variables declared with the static storage class specifier have static extent (as opposed to local extent, signified by auto). An object is said to have static extent when it is allocated storage at or before the beginning of program execution and the storage remains allocated until program termination. Consider the following example:
#include <stdio.h>
      void foo(int *p)
            static int k = 5;
            *p += k;
      int main(void)
            int k = 0, *p = &k;
            printf(“k = %d\n”, k);
            printf(“k = %d\n”, k);
            printf(“k = %d\n”, k);
The three values displayed by the program would be 6, 13, and 21, respectively. Note that object k with block-scope in function foo has static extent and is initialized with the integer value 5 at or before the beginning of program execution
Your submission would consist of two files: a C source file and a header file.
Your submission must not use any global variables.
 No Standard C Library functions are to be used in your submission
Ensure that there are no statements that perform I/O.
Question by:postal21
  • 2
LVL 45

Expert Comment

ID: 9777380
sorry postal21 ... EE membership agreement forbids us from doing your homework ...
You have to do it on your own ...
If you are stuck with a specific problem, we can give you a push but we cannot do it for you
LVL 45

Expert Comment

ID: 9777412
that been said, perhaps you may find strcspn() function useful

man strcspn for more details
LVL 22

Accepted Solution

grg99 earned 500 total points
ID: 9782242
We can't do the whole thing, but here's a  outline.
Note , several boundary conditions and off-by-ones left in as
an exercise. :)

char * p = NULL;

char* StringToken (char *str, const char *set)
if( str != NULL ) p = str;

Start = p;
while( Isin( *p++, set ) ) p++;
*p = '\0';
return( Start );

int Isin( char Target, cha * List )
int Found;
Found = 0;
while( ! Found && *List != '\0' ) Found = *List++ == Target
return( Found );


Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Preface I don't like visual development tools that are supposed to write a program for me. Even if it is Xcode and I can use Interface Builder. Yes, it is a perfect tool and has helped me a lot, mainly, in the beginning, when my programs were small…
This is a short and sweet, but (hopefully) to the point article. There seems to be some fundamental misunderstanding about the function prototype for the "main" function in C and C++, more specifically what type this function should return. I see so…
The goal of this video is to provide viewers with basic examples to understand how to use strings and some functions related to them in the C programming language.
The goal of this video is to provide viewers with basic examples to understand and use switch statements in the C programming language.

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question