Strtok function

Posted on 2003-11-19
Last Modified: 2012-06-27
If anyone writes this before Saturday and it passes my main.c function test (can be requested) it’s worth 500 points.

The motivation behind this program is to implement the functionality of the tokenizer that is a part of all compilers, interpreters, parsers, and scripting languages. For example, the C parser identifies the following comma-separated list of tokens (which are the smallest elements of a C program that are meaningful to the compiler) from the expression a=i+++j: a, =, i, ++, +, j. For our program, we will use the tokenizer to interpret the contents of a null-terminated string as words by filtering out the non-alphabetical characters. The following is a description of the function that you must implement:
Write a function StringToken() with the following prototype:
char* StringToken (char *str, const char *set);
StringToken mimics the behavior of function strtok declared in the C standard library file string.h. The explanation for StringToken is a function that searches str for tokens delimited by characters from set. A sequence of calls of StringToken(str, set) splits str into tokens, each delimited by a character from set. The first call in a sequence has a non-null str. It finds the first token in str consisting of characters not in set; it terminates that by overwriting the next character of str with ‘\0’ and returns a pointer to the token. Each subsequent call, indicates by a null value of str, returns the next such token, searching from just past the end of the previous one. StringToken returns null when no further token is found. The string set may be different on each call.
Sample Program:
Here is a sample program that illustrates the use of StringToken to determine the number of words in a null-terminated string by filtering out certain non-alphabetical characters:
#include <stdio.h>
#define LINE_LEN            1024
char * StringToken(char *str, const char *set)
            /* … */
int main(void)
      Note the use of the escape sequence \" to represent double quotation marks within
the string. An escape sequence begins with the backslash '\' and is used to represent certain nonprintable characters ( for example: '\0', '\a' (bell)),
the single and double quotation marks and the backslash characters ('\'', '\"', '\\').
            char line[LINE_LEN] = “\”My goodness,\” she said, \”Is that right?\””;
            /* Set of non-alphabetic characters */
char set[] = “ .,?!’\”\t\n”;
            char *word = StringToken(line, set); /* find first word */
            while (word)
                  printf("<%s>\n", word);
                  word = StringToken(0, set); /* find next word */
            return 0;
The results of the above program are the words (delimited by <>):
The function StringToken requires a variable defined with the storage-class specifier static. Block-scope variables declared with the static storage class specifier have static extent (as opposed to local extent, signified by auto). An object is said to have static extent when it is allocated storage at or before the beginning of program execution and the storage remains allocated until program termination. Consider the following example:
#include <stdio.h>
      void foo(int *p)
            static int k = 5;
            *p += k;
      int main(void)
            int k = 0, *p = &k;
            printf(“k = %d\n”, k);
            printf(“k = %d\n”, k);
            printf(“k = %d\n”, k);
The three values displayed by the program would be 6, 13, and 21, respectively. Note that object k with block-scope in function foo has static extent and is initialized with the integer value 5 at or before the beginning of program execution
Your submission would consist of two files: a C source file and a header file.
Your submission must not use any global variables.
 No Standard C Library functions are to be used in your submission
Ensure that there are no statements that perform I/O.
Question by:postal21
  • 2
LVL 45

Expert Comment

ID: 9777380
sorry postal21 ... EE membership agreement forbids us from doing your homework ...
You have to do it on your own ...
If you are stuck with a specific problem, we can give you a push but we cannot do it for you
LVL 45

Expert Comment

ID: 9777412
that been said, perhaps you may find strcspn() function useful

man strcspn for more details
LVL 22

Accepted Solution

grg99 earned 500 total points
ID: 9782242
We can't do the whole thing, but here's a  outline.
Note , several boundary conditions and off-by-ones left in as
an exercise. :)

char * p = NULL;

char* StringToken (char *str, const char *set)
if( str != NULL ) p = str;

Start = p;
while( Isin( *p++, set ) ) p++;
*p = '\0';
return( Start );

int Isin( char Target, cha * List )
int Found;
Found = 0;
while( ! Found && *List != '\0' ) Found = *List++ == Target
return( Found );


Featured Post

What Security Threats Are You Missing?

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

An Outlet in Cocoa is a persistent reference to a GUI control; it connects a property (a variable) to a control.  For example, it is common to create an Outlet for the text field GUI control and change the text that appears in this field via that Ou…
Preface I don't like visual development tools that are supposed to write a program for me. Even if it is Xcode and I can use Interface Builder. Yes, it is a perfect tool and has helped me a lot, mainly, in the beginning, when my programs were small…
The goal of this video is to provide viewers with basic examples to understand and use pointers in the C programming language.
The goal of this video is to provide viewers with basic examples to understand opening and reading files in the C programming language.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now