How to write a scanner program for a compiler?

I have this simple program saved in a text file:
{ Sample program
  in TINY language-
  computes factorial
read x; { input an integer }
if 0 < x then { don't compute if x <= 0 }
  fact := 1;
    fact := fact * x;
      x := x - 1
      until x = 0;
      write fact { output factorial of x }

I need to figure out how to write a program that will read this file, pick out the tokens, and then write the tokens only to another text file.  Please, can anyone give me some insight on how to even start this project!!!
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jaime OlivaresSoftware ArchitectCommented:
Is a long question. You have to learn some about parsing techniques, first.
Have a look to these articles:
String splitting:
Simple math parser:
Math parser:
Advanced C++ parser:

for the above simple project you can use lex ,

Do you mean that?

#include <vector>
#include <string>
#include <iostream>
#include <fstream>

using namespace std;

int main(int nArgs, char* pszArgs[])
      vector<string> tokens;
      string token;
      ifstream ifs("input.txt");
      while (!
           ifs >> token;
           if ( || token.empty())


      ofstream ofs("output.txt");
      for (int i  = 0; i < tokens.size(); ++i)
           ofs << tokens[i] << endl;
       return 0;

Regards, Alex
Cloud Class® Course: CompTIA Healthcare IT Tech

This course will help prep you to earn the CompTIA Healthcare IT Technician certification showing that you have the knowledge and skills needed to succeed in installing, managing, and troubleshooting IT systems in medical and clinical settings.

First we need to know if you're supposed to write the scanner from scratch, or whether you are allowed to use any canned tokenizer tools, like lex, yacc, or token objects.

If you have to write it from scratch, here's the basic outline:

   get next input character

  if it is a left curly brace:  repeat getting next character until EOF or you find a right curly brace.
  if it is a digit:  keep getting input characters until you get a non-digit: return the digits as a kind "number"
  if it is a letter:  keep getting input characters until you get a non-letter: return the letters as a kind "word"
   if it is a space, tab, or end-of-line:  keep getting characters until you get something that isnt a space, tab, or end of line.
   if it is a colon:  get the next character, if it is a equals, return kind = "assignment operator", otherwise return "error"
   if it is anything else: return the character as kind = "operator"
   get next character
 until EOF

morales7_0Author Commented:
I'm sorry fellows, the program is from scratch and it has to be written in C++.  I should have wrote that in my details of the question.  For now, the program should only be able to recognize the tokens.  The next program I have to do later on is going to be the parse program and I have implement this scan program with it.  
LCC has a implementation of scanner and a parser.

Its in C though , so you have to read andunderstand and then make a C++ version out of it.
The source code is available from the website, but its preferable to get the book too. The book is
quite nice to read bout practical computer implementation vis a vis the theory books like the dragon book.
From scratch, eh?  Then you could use my outline above, easily translated into C or C++ code. Less than a page of code.


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.