?
Solved

How to find the character encoding type of a file in Linux?

Posted on 2011-03-21
7
Medium Priority
?
916 Views
Last Modified: 2012-05-11
I have a .txt file and I need to determine what character encoding it is using so I can then convert other files to match it.

If I run "file myfile.txt", I get this info:
       "Non-ISO extended-ASCII text, with very long lines"

I know the file is ANSI but I need to determine exactly what type of ANSI file so I can convert other files to match it.

When I check the filetypes available in "iconv", I find these possibilities. How do I determine which one is the exact match?

ANSI_X3.4-1968
ANSI_X3.4-1986
ANSI_X3.4
ANSI_X3.110-1983
ANSI_X3.110
ASCII
MS-ANSI
WINDOWS-31J
WINDOWS-874
WINDOWS-936
WINDOWS-1250
WINDOWS-1251
WINDOWS-1252
WINDOWS-1253
WINDOWS-1254
WINDOWS-1255
WINDOWS-1256
WINDOWS-1257
WINDOWS-1258


0
Comment
Question by:bearclaws75
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 5
  • 2
7 Comments
 
LVL 31

Expert Comment

by:farzanj
ID: 35185608
Just use the command
unix2dos filename


And it should convert it to the DOS format.
or sometimes called
ux2dos
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35185616
If you want to go the other way,

issue this command

dos2unix filename
0
 
LVL 31

Expert Comment

by:farzanj
ID: 35185634
I think the character encoding is UTF-8
0
Get free NFR key for Veeam Availability Suite 9.5

Veeam is happy to provide a free NFR license (1 year, 2 sockets) to all certified IT Pros. The license allows for the non-production use of Veeam Availability Suite v9.5 in your home lab, without any feature limitations. It works for both VMware and Hyper-V environments

 

Author Comment

by:bearclaws75
ID: 35185695
farzani - I know the file is not UTF-8 because if I run "file otherfile.txt" on a different file, the output is:
     "UTF-8 Unicode text, with very long lines, with CRLF line terminators"

Howver, I ran "unix2dos myfile.txt" and it converted the file:
     "unix2dos: converting file myfile.txt to DOS format ..."

...but if I run "file myfile.txt", I get the same info:
       "Non-ISO extended-ASCII text, with very long lines"

"unix2dos" is a good command-line utility but, ultimately, i need to determine the exact character encoding so I can update my php scripts to generate the proper file type.


0
 
LVL 31

Expert Comment

by:farzanj
ID: 35185797
Well, I see your point but you can still call this utility from within PHP.  In any case let me look into it
0
 
LVL 31

Accepted Solution

by:
farzanj earned 1500 total points
ID: 35185912
Well, I think it is very simple.  Basically you are converting the new line characters, that is about all.  Rest the remaining are the ASCII codes for characters which are the same.

So you need to convert line feed (\n) to carriage return (\r) and line feed.  Use a simple regular expression to do that.

So you are changing \n  to \r\n
0
 

Author Closing Comment

by:bearclaws75
ID: 35217944
I found this command which did the trick:

sed 's/\r$//' winfile.txt > unixfile.txt

I still wasn't able to determine the *exact* file encoding but this produced the desired results.

Thanks for the help!
0

Featured Post

Containers & Docker to Create a Powerful Team

Containers are an incredibly powerful technology that can provide you and/or your engineering team with huge productivity gains. Using containers, you can deploy, back up, replicate, and move apps and their dependencies quickly and easily.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Using dates in 'DOS' batch files has always been tricky as it has no built in ways of extracting date information.  There are many tricks using string manipulation to pull out parts of the %date% variable or output of the date /t command but these r…
VALIDATING DATES One method of validating dates is to jam the date into the DATE command and see if it accepts it by examining the system's errorlevel value. A non-zero result indicates failure. A typical example might look something like the fol…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…
Visualize your data even better in Access queries. Given a date and a value, this lesson shows how to compare that value with the previous value, calculate the difference, and display a circle if the value is the same, an up triangle if it increased…
Suggested Courses
Course of the Month12 days, 9 hours left to enroll

777 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question