We help IT Professionals succeed at work.

Tokenizing words in a file

Vlearns
Vlearns used Ask the Experts™
on
1 #include <fstream>
  2 #include <sstream>
  3 #include <iostream>
  4
  5 using namespace std;
  6
  7 int main() {
  8     ifstream file("data1");
  9     string line;
 10     stringstream line_stream;
 11     string word;
 12
 13     while (getline(file,line)) {
 14        line_stream << line;
 15
 16        while (getline(line_stream, word,' ')){
 17            cout << word << ":";
 18        }
 19
 20        cout << endl;
 21        cout << line_stream.str() << endl;
 22        line_stream.str("");
 23     }
 24 }

where data 1:

the fox is quick
the sky is blue
hello world

what i am expecting is

the:fox:is:quick
the:sky:is:blue
hello:world

but what i see is  just

the:fox:isquick

what am i doing wrong?

i want to read strings from a file and tokenize them.

thanks
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
I have never tried getline() with stringstream this way (namely with the space as a delimiter). So, I may not give you a satisfactory answer on what is wrong with your code.

The more usual way in the case is to tokenize the line using the standard strok() function.  However, it is destroys the line.  You can try the more flexible stringtok() template from here http://gcc.gnu.org/onlinedocs/libstdc++/manual/strings.html#strings.string.token

Do insist on explanation of the behaviour of your code.  I hope some better expert explains that.
The problem is that after first iteration the line_stream is in an EOS (end of stream) state and therefore further operations fail. This is one of the nuisances of STL streams that they fail silently and many times you are left to wonder where what went wrong.

Use this

int main()
{
    ifstream file("F:\\Project\\matrix\\Debug\\data1.txt");
    string line;
    stringstream line_stream;
    string word;

    while (getline(file,line))
      {
       line_stream << line;

       while (getline(line_stream, word,' ')){
           cout << word << ":";
       }

       cout << endl;
       cout << line_stream.str() << endl;
       line_stream.str("");
       line_stream.clear();
    }
}

OR BETTER

int main()
{
    ifstream file("F:\\Project\\matrix\\Debug\\data1.txt");
    string line;
    string word;

    while (getline(file,line))
    {
       stringstream line_stream;
       line_stream << line;

       while (getline(line_stream, word,' ')){
           cout << word << ":";
       }

       cout << endl;
       cout << line_stream.str() << endl;
    }
}

Author

Commented:
yes , the missing clear was the problem