Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 626
  • Last Modified:

how to deal with big text files.

The problem:
- We have a big text files.
- a text file of bad words

The result:
I'd like to have a new text files without all bad words

any ideas ???
0
us111
Asked:
us111
1 Solution
 
johnsmCommented:
Read in the file, line by line. As you read in each line, check it for "bad" words. If there are any, then eliminate them and write the data back to a new file with the bad words having been removed.

What are you calling bad words? Curse words, or words that are improperly spelled or formatted?
0
 
us111Author Commented:
- please give me a working example

- bad words: I mean words like 'a' , 'the'....
0
 
johnsmCommented:
# Open the data file for reading
# ------------------------------
  open (FILE, "$dirpath\\$file");

# Open a second file for writing
# ------------------------------
  open (FILE2, ">$dirpath\\$file");

# Read through it one line at a time
# ----------------------------------
  while ($dataline = <FILE>) {

# Replace the bad words with nothing
# ----------------------------------
  $dataline =~ s/a//ig;
  $dataline =~ s/the//ig;

# or do it all on one line
# ------------------------
  $dataline =~ s/(a|the|whatever|whatever)//ig;

# Write the line back to the new file
# -----------------------------------
  print FILE2 "$dataline\n";

# End your loop
# -------------
  }

# Close both files
# ----------------
  close(FILE);
  close(FILE2);

# The end
# -------
  end;

0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
johnsmCommented:
You probably won't need the \n on the line

print FILE2 "$dataline\n";

The line should already contain a new line character. So maybe just

print FILE2 "$dataline";

will work for you.


0
 
johnsmCommented:
oops. One other mistake I now see you can't open the same file for both reading and writing.

Change

# Open a second file for writing
# ------------------------------
 open (FILE2, ">$dirpath\\$file");

to

# Open a second file for writing
# ------------------------------
 open (FILE2, ">$dirpath\\$file2");

sorry.


0
 
smiskCommented:
I think you're going to want to put word boundaries on that, otherwise all occurences of "a" will be removed (ie, "that" will become "tht").

Change the replace lines to the following format :

s/(\W|^)a(\W|$)/ /g;

where "a" is what you want to remove.
0
 
ozoCommented:
#or
s/\b(a|the|whatever|whatever)\b//ig;
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now