perl: How dose perl handle diacritics and non roman characters

I'm trying to ingest a large text file into PostgreSQL Full details here

I may need to run  a script against the file to either remove the problem rows into a separate file, or insert into the database via DBI and was wondering how perl handles diacritics?

I'm sure perls native I/O file handle is UTF8 but not certain about the diacritics or non roman characters
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

It depends what version of Perl you are using.  Assuming the data is Unicode, I'm pretty sure you are fine if you are using 5.8 (maybe 5.10) or later - 5.6 definitely had some Unicode bugs.

I assume DBD::Postgres handles Unicode fine (if PostgreSQL does) but I don't use it so I'm not sure.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
trevor1940Author Commented:
Assuming the data is Unicode

Yes the docs state Unicode UTF8


perl -v

perl 5 version 20 (v5.20.2) for WIN32 -x64-multi-thread

Open in new window

There is so much information regarding perl and charsets I'm now confused

Any suggestions how to test if a script is corrupting the text?
bearing in mind i'm running in a windows command prompt which is rubbish at displaying extended characters
trevor1940Author Commented:
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.