Solved

problem with using strtok()

Posted on 2004-09-17
8
490 Views
Last Modified: 2008-03-04
strtok() treats ,, (2 commas) as , (1 comma), how can i display empty character between ,,  ? for example

I have a string like " abc,def,ghi,,jkl" (5 columns), the output format should be abc def ghi   jkl ( 5 columns)
but strtok() displays abc def ghi jkl (only 4 columns)

Whats the fix for this type of problem ?
0
Comment
Question by:justinY
8 Comments
 
LVL 86

Expert Comment

by:jkr
Comment Utility
Just check the lenght of the token that is returned - if it is 0, you have to consecutive delimiters and need to handle that properly.
0
 
LVL 19

Expert Comment

by:drichards
Comment Utility
No, strtok skips leading delimiters, so if you have a bunch of delimiters in a row, they are all skipped - there is no such thing as a zero-length token.

From  Linux Programmer's Manual:

"The strsep() function was introduced as a replacement for strtok(), since the latter cannot handle empty fields. However, strtok() conforms to ANSI-C and hence is more portable."

You will need to implement your own strtok function or use strsep if your platform supports it.  Really cheap version of custom strtok is given below.  

char *MyStrTok(char *text, const char del)
{
    static char *curTok = NULL;
    char *result = NULL;
    if ( text != NULL )
    {
        curTok = text;
    }
    result = curTok;
    if ( curTok != NULL )
    {
        curTok = ::strchr(curTok, del);
        if ( curTok != NULL ) *curTok++ = 0;
    }

    return result;
}
0
 
LVL 39

Expert Comment

by:itsmeandnobodyelse
Comment Utility
Or that:



#include <iostream>
#include <string>
#include <vector>

using namespace std;

int stringToArray(const string& str, char delim, vector<string>& strArr)
{
    size_t next;
    size_t last = 0;
    while ((next = str.find(delim, last)) != string::npos)
    {
        strArr.push_back(str.substr(last, next - last));
        last = next + 1;
    }
    strArr.push_back(str.substr(last));
    return strArr.size();
}


int main(int argc, char* argv[])
{
     string str = ",abc,def,,ghi,jki,";
     vector<string> strArr;
     int nTok = stringToArray(str, ',', strArr);
     cout << str << " # of tokens " << nTok << endl;
     
     for (int i = 0; i < nTok; i++)
         cout << strArr[i] << endl;

     getline(cin, str);

     return 1;
}

Regards, Alex
0
 

Author Comment

by:justinY
Comment Utility
Thank you very much to all
I have Microsoft VC++.Net standard version. It doesnt support strsep(). So I use custom version strtok() that is MyStrTok() given by drichards. But I am having compiling error. I tried many times, just cannot solve the problem. Can someone take a look at it ? Thank you
Here is my code.

#include <iostream>
#include <fstream>
#include <iomanip>
#include <string>

using namespace std;

const int MaxSize=1024;

//////////////////////////////////
//
// struct for reference data
//
//////////////////////////////////
typedef struct
{
 char keyData[10];
 char valueData[20];
}KeyRef;


//////////////////////////////////////////
// function to lookup value by key
//////////////////////////////////////////
char* findKeyValue(KeyRef* ref, char* k){
      
      for (int i=0;i<MaxSize;i++){
            if (!strcmp(ref[i].keyData,k))
                  return ref[i].valueData;
      }
      return "None";
};


///////////////////////////////
//
// Custom version of strtok()
//
///////////////////////////////

char* MyStrTok(char* text, const char del)
{
    static char *curTok = NULL;
    char *result = NULL;
    if ( text != NULL )
    {
        curTok = text;
    }
    result = curTok;
    if ( curTok != NULL )
    {
        curTok = ::strchr(curTok, del);
        if ( curTok != NULL ) *curTok++ = 0;
    }

    return result;
};

///////////////////////////////
//
// main
//
///////////////////////////////
main(int arc, char *arv[])
{

ifstream fin("fin.txt");
ofstream fout("fout.txt");
char row_read[512];
char row_elem[10][50];

ifstream fin2("frefer.txt");
ofstream fout2("refer_out.txt");
KeyRef refer_data[MaxSize];
char refer_read[MaxSize];
char refer_row_elem[2][MaxSize];
int i=0,j=0,k=0;

//////////////////////////////////////////
// read refer file into struct of array
//////////////////////////////////////////
while( fin2.getline( refer_read,MaxSize ))
{
      char *tok = strtok(refer_read,",");
      while( tok != NULL) {
            strcpy(refer_row_elem[j],tok);
            tok= strtok(NULL,",");
            j++;
      }
      strcpy(refer_data[k].keyData, refer_row_elem[0]);
      strcpy(refer_data[k].valueData, refer_row_elem[1]);
      j=0;
      k++;
}

/////////////////////////////////////////////////
// output refer file , review data only
/////////////////////////////////////////////////
//for (int q=0;q<k;q++){
//       fout2 << setw(10) << refer_data[q].keyData << setw(10) << refer_data[q].valueData << endl;
//}


///////////////////////////////////////////////////////////////////
// for each line of file  
// assume there are 5 collums in each row in file fin.txt
///////////////////////////////////////////////////////////////////
while( fin.getline( row_read, sizeof( row_read ) ) )
{
      //////////////////////////////////////////
      // parse string with ","
      //////////////////////////////////////////
      char *token = MyStrTok( row_read, "," );
      //char *token = strsep( row_read, "," );
      while( token != NULL )
      {
        /* While there are tokens in "string" */
        strcpy(row_elem[i], token );
        /* Get next token: */
        token = strtok( NULL, "," );
        i++;
      }

      //////////////////////////////////////////////////////////////////////
      // get reference value by looking at key in reference file
      //////////////////////////////////////////////////////////////////////
      char* valueFound;
      valueFound = findKeyValue(refer_data, row_elem[0]);
      cout << " value found - " << valueFound << endl;

      //////////////////////////////////////////////////////////////////////
      // output to file, first colmn replaced with value from reference file
      //////////////////////////////////////////////////////////////////////
      fout<<setw(10)      << valueFound
          <<setw(10)  << row_elem[4]
          <<setw(10)      << row_elem[3]
          <<setw(10)      << row_elem[2]
          <<setw(10)      << row_elem[1]
          <<endl;

      i=0;
}

return 0;
}
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 19

Expert Comment

by:drichards
Comment Utility
1) Make sure you turn off precompiled headers for this file or add '#include "stdafx.h"' at the top of the file.

2) '     char *token = MyStrTok( row_read, "," );' should be:  char *token = MyStrTok( row_read, ',' );
Note the single quotes - my version of strtok takes a single character as a delimiter.  The code is a lot simpler if you don't need multiple delimiters.

If you need to turn off precompiled headers, right click on the source file in the Solution Explorer and select "Properties". Go to the C/C++->Precompiled Headers page and choose "Not Using Precompiled Headers" on the "Create/Use Precompiled Header" line.
0
 

Author Comment

by:justinY
Comment Utility
drichards,
single quotes works, but other problems.
1. #include "stdafx.h" cannot be found in my vc7\include folder.
2. precompiled headers have been turn off.
3. without '#include "stdafx.h"', I can compile my code, but the output is funny. It only catch one column.
I post my fin, and frefer files. you run the code and will see the output.

fin:
id,notes1,notes2,notes3,account name,amount
111,aaa,a3,a4,,100000
222,bbb,b3,b4,,200000
333,ccc,c3,c4,cccc5,300000
444,ddd,d3,d4,dddd5,400000
555,eee,e3,e4,eeee5,500000
666,fff,f3,f4,ffff5,600000
777,ggg,g3,g4,gggg5,700000
888,hhh,h3,h4,hhhh5,800000
999,iii,i3,i4,iiii5,900000


frefer:
111,111aaa
222,222bbb
333,333ccc
444,444ddd
555,555eee
666,666fff
777,777ggg
888,888hhh
999,999iii


Thanks and waiting online
0
 
LVL 19

Accepted Solution

by:
drichards earned 500 total points
Comment Utility
While parsing fin, you use MyStrTok to start but then use strtok in the loop.  Change:

       token = strtok( NULL, "," );
to
       token = MyStrTok( NULL, ',' );

in the 'while (fin.getline(...))' loop.

There is also a minor logic error in that you try to look up the header line info and get a result of 'None' which is prined out as the header if the ID column.  You'll want to fix that.  Probably the first time through (this will be the header line) don't do 'findKeyValue'.
0
 
LVL 55

Expert Comment

by:Jaime Olivares
Comment Utility
Here is an implementation of strsep() function:
http://www.greatsnakes.com/Sepal/d4/d0/strsep_8c-source.html
0

Featured Post

Find Ransomware Secrets With All-Source Analysis

Ransomware has become a major concern for organizations; its prevalence has grown due to past successes achieved by threat actors. While each ransomware variant is different, we’ve seen some common tactics and trends used among the authors of the malware.

Join & Write a Comment

  Included as part of the C++ Standard Template Library (STL) is a collection of generic containers. Each of these containers serves a different purpose and has different pros and cons. It is often difficult to decide which container to use and …
Introduction This article is a continuation of the C/C++ Visual Studio Express debugger series. Part 1 provided a quick start guide in using the debugger. Part 2 focused on additional topics in breakpoints. As your assignments become a little more …
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now