Solved

Can one use Ruby to rename a filename encoded with Windows-1252 format, containing special characters that do not convert to UTF-8, on Windows XP?

Posted on 2011-03-16
9
963 Views
Last Modified: 2012-05-11
Long question, but that's the deal.

I'm sorting EML files.  I have a program that scans directories to pull out EMLs, and another that sorts them back into a proper structure. (One I want)  That's working fairly well.  Or, it was.

Until I got a ton of eml files from a user, with names created from subject lines of the email.  This would be fine, if he apparently didn't have a ton of spam with special characters. (No idea what it was, may Cyrillic, Chinese, Japanese. Not sure, doesn't matter)  This made Ruby very unhappy.

I'm using Ruby 1.9+, so it will read the file just fine. The problem seems to be that Ruby naturally takes these files and displays them differently. I believe it's converting the original name, which was Windows-1252 into UTF-8. Which would be fine, but when I go to rename the file, Ruby can't find the file, because the conversion changed the filename.  

The analog would be you telling me I just got a new employee named Jorge.  I decide the employee is actually named 'George'.  But when I go to work with 'George', there isn't anyone there named George, but Jorge still exists.  The file is there, Ruby just changed the working name.

My problem is that I can't figure out how to Ruby to cut it out.  There are several tips on how to encode the data within a file, but little about the file name, namely because I can't reference the file to modify it.  And I cannot encode the filename variable to Windows-1252, because it throws an exception regarding characters in UTF-8 that will not convert to Windows-1252. (Which makes me question how it went from Windows-1252 to UTF-8 in the first place; perhaps it's a one-way conversion with substituted letters.)

I can't find a way around this, short of moving the EML files (many gigs) over to a Linux based machine, running the script there, and then bringing them over to the Windows machine for final sorting.  I'm hoping that will work, but I'm questioning if there is a better way around this.
FileUtils.move pwd+"/"+f1, $scriptDir+"/"+f1

Open in new window

0
Comment
Question by:FoxKeegan
  • 6
  • 2
9 Comments
 
LVL 5

Expert Comment

by:paulqna
ID: 35152555
This will allow you to choose source AND destination filename for the file you are converting...
0
 

Author Comment

by:FoxKeegan
ID: 35158269
Have you worked with this gem in the past?  On Windows for that matter?

RubyGems installed; the gem installed but "require 'chilkat'" fails.  Maybe the libs are screwed up.

Going to try on another Windows box. That failing, I'll try it on a Linux box. (both before and after installing Chilkat with the linux box)

Failing all that, I'll go back and figure out why the gem isn't being called correctly. (Undoubtedly something I'm doing wrong.)
C:\Ruby192>gem list chilkat -d

*** LOCAL GEMS ***

chilkat (9.1.2)
    Platform: x86-mswin32-60
    Author: Chilkat Software, Inc.
    Homepage: http://www.chilkatsoft.com/ruby.asp
    Installed at: C:/Ruby192/lib/ruby/gems/1.9.1

    Chilkat Ruby library

C:\Ruby192>ruby -rubygems -e 'require "chilkat"'
C:/Ruby192/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require':
no such file to load -- chilkat (LoadError)
        from C:/Ruby192/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:i
n `require'
        from -e:1:in `<main>'

Open in new window

0
 
LVL 5

Expert Comment

by:paulqna
ID: 35159645
I cannot help you.
0
 

Author Comment

by:FoxKeegan
ID: 35160834
At present the easiest solution is to use a command line bulk renaming utility (freeware) on the directory structure, and then running the Ruby scripts to perform the tasks. (Anything not renamed properly is skipped to avoid errors)

Wish I had a better solution.  Perhaps writing it in JAVA would work better...
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 4

Expert Comment

by:kristinalim
ID: 35162216
As far as I know, gems need to be activated before requiring (to add them to require paths) in Ruby 1.9.1. I haven't tested with the chilkat gem, but I do similar for other gems:

ruby -rubygems -e "gem 'chilkat' ; require 'chilkat'"

Open in new window

0
 

Author Comment

by:FoxKeegan
ID: 35179853
Well I believe I've deduced why Chilkat isn't working.  Thank you for that Kristin. The error I was given after attempting to activate the gem got me thinking and I went to check Chilkat's website a bit more in depth:


Important:

The Chilkat Ruby module works with Ruby v1.8.*. It is not yet compatible with Ruby v1.9.*.

So that explains that.

I believe all my code should work with version 1.8 so I'll give that a shot.
C:\Ruby192>ruby -rubygems -e "gem 'chilkat' ; require 'chilkat'"
C:/Ruby192/lib/ruby/site_ruby/1.9.1/rubygems.rb:900:in `report_activate_error':
RubyGem version error: chilkat(9.1.2 not >= 0) (Gem::LoadError)
        from C:/Ruby192/lib/ruby/site_ruby/1.9.1/rubygems.rb:248:in `activate'
        from C:/Ruby192/lib/ruby/site_ruby/1.9.1/rubygems.rb:1276:in `gem'
        from -e:1:in `<main>'

Open in new window

0
 

Author Comment

by:FoxKeegan
ID: 35193647
Ruby isn't horribly keen on installing older versions.  Trying to get 1.8.7 working on Linux was...fun, but I can't even find information regarding which version of RubyGems works with 1.8.7, and the most current certainly doesn't.  In fact, I can't even find an older version of RubyGems for linux on their website. They just state you should either use apt-get, yum or link to the SVN. (CentOS doesn't have RubyGems in its repos and at this point I'm not reinstalling Fedora to check.)

I've tried so many routes to get this functional that I'm having to call it quits. It's easier to let the script do the work, have me run another utility to rename the files that didn't convert later, and then run it again.

I'm still skeptical the latest version of Ruby can't do something as simple as renaming a file called "‚¨Œ³‹C‚Å‚·‚©H_66_20101111_130027_093.eml" to "002381.eml"  I'm sure I'm just missing something.

You can bet next time I'll just write the program in JAVA.
0
 

Accepted Solution

by:
FoxKeegan earned 0 total points
ID: 35202651
At this time of writing, to install an old version of Ruby with an appropriate version of RubyGems that will work with Chilkat on the 64-bit version of CentOS 5.5, see the attached code.

Then simply include:

require "rubygems"
require "chilkat"

...within your code.  

Unfortunately I've no intention of finding out if this gem will actually accomplish that which I set out for when asking this question. I've spent too much time on this already and chilkat's incompatibility with the latest version of Ruby makes it undesirable, and less portable.

With luck, someone will find this useful.
yum install openssl openssl-devel gcc

wget http://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p302.tar.gz
tar -zxvf ruby-1.8.7-p302.tar.gz
cd ruby-1.8.7-p302
./configure --with-openssl-dir=/usr/lib/openssl
make
make install
cd ..
wget http://production.cf.rubygems.org/rubygems/rubygems-1.3.7.tgz
tar -zxvf rubygems-1.3.7.tgz
cd rubygems-1.3.7
ruby setup.rb config
ruby setup.rb setup
ruby setup.rb install
cd ..
wget http://www.chilkatsoft.com/download/chilkat-9.1.2-x86_64-linux.gem.tar.gz
tar -zxvf chilkat-9.1.2-x86_64-linux.gem.tar.gz
gem install chilkat-9.1.2-x86_64-linux.gem

Open in new window

0
 

Author Closing Comment

by:FoxKeegan
ID: 35230130
Selected my own solution as no other solution was more complete based upon the initial question.

Grade letter lowered due to incomplete solution.
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

In Ruby, Call or invoke a API DLL library is easily via Win32API class, win32-api gem or other gems. For general DLL API call, there are quite a few references, some good tips list below: http://www.rubytips.org/2008/05/13/accessing-windows-api-fro…
Recently I spent hours debugging an issue in a Rails project where ActiveRecord was causing MySQL errors trying to create a User object of a class at the top level of a Single Table Inheritance model structure.  It turns out `.create` behaves differ…
Here's a very brief overview of the methods PRTG Network Monitor (https://www.paessler.com/prtg) offers for monitoring bandwidth, to help you decide which methods you´d like to investigate in more detail.  The methods are covered in more detail in o…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now