Special characters in DB2 won't import to SQL via CSV file

I have a large table in DB2 (>1GB) that I need to export to a CSV and then import to SQL Server 2008.  Security rules prohibit a direct connection to DB2 from SQL, so I must export to the CSV file.

There are special characters in the DB2 table that prevent the SQL import to succeed.  Is there a way to identify these characters (I have even seen an arrow character), and replace them with something that will import properly?
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jim HornMicrosoft SQL Server Developer, Architect, and AuthorCommented:
>Is there a way to identify these characters
Give this a whirl..


Download a trial of the super-dooper text editor Ultra Edit.


Open it, load your file.


Hit ctrl-H to go into Hex mode, which shows two windows.
Left side is your file, including any characters that would not be visible in a .csv.  
Right side is the Hex number of every character on the left side.


Do a google search for 'ANSI character set' and eventually you'll find one that maps out every character's value, ANSI, and HEX.


Now you know what characters you'll need to weed out, and edit your ETL package accordingly.
pansophyAuthor Commented:
Jim, I got to step 4 without a problem.  But how do I know which characters will give SQL a problem?

Also am I correct that once I identify these characters i am doing a HEX find/replace?
Kent OlsenDBACommented:
Hi pansophy,

What character sets (code pages) are you using for DB2 and SQL Server?
The 7 Worst Nightmares of a Sysadmin

Fear not! To defend your business’ IT systems we’re going to shine a light on the seven most sinister terrors that haunt sysadmins. That way you can be sure there’s nothing in your stack waiting to go bump in the night.

pansophyAuthor Commented:
It appears that DB2 is using this character set "IBM All Character 437 Set (ANSI)" to create the CSV file.  

Kent OlsenDBACommented:
What version of DB2 are you running?  Z/OS, UDB on Windows, UDB on *nix, or AS400?

On most systems you can run the db2 command to find the code page setting

     db2 get db cfg for {databasename}

The code page, code set, and collating information are in the first 8 or 10 lines.
pansophyAuthor Commented:
Database configuration release level                    = 0x0f00
Database release level                                  = 0x0f00
Database territory                                      = US
Database code page                                      = 819
Database code set                                       = ISO8859-1
Database country/region code                            = 1
Database collating sequence                             = UNIQUE
Alternate collating sequence              (ALT_COLLATE) =
 Number compatibility                                    = OFF
Varchar2 compatibility                                  = OFF
Date compatibility                                      = OFF
Kent OlsenDBACommented:

819 is the "normal" code page used when DB2 is running on AIX.  The key to making completing this conversion is to knowing what character set the SQL Server database is using.  If it's utf-8 (a reasonable choice) you can get the correct export with something like this:

 db2 "export to mytable.csv of del modified by CODEPAGE=1208 select * from mytable"


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today

From novice to tech pro — start learning today.