SQL Server XML Format File

What is the best way to define MAX_LENGTH on a VARCHAR datatype that can exceed 8000 characters in an XML format file?
If you do not use the MAX_LENGTH attribute, MS states the default is 8000 characters. We have some cases (not many) where the length can be considerably more than that. I don't want to go sticking 1000000 in there when I don't know what the impact will be of importing tens of thousands of rows of data.
LVL 15
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Pawan KumarDatabase ExpertCommented:
Please use VARCHAR(MAX)

it can store up to 2GB of data.

From Microsoft.

varchar [ ( n | max ) ]
Variable-length, non-Unicode string data. n defines the string length and can be a value from 1 through 8,000. max indicates that the maximum storage size is 2^31-1 bytes (2 GB). The storage size is the actual length of the data entered + 2 bytes. The ISO synonyms for varchar are char varying or character varying.


if we provide size , we can go upto 8000 characters
else if we use Max then we can go upto 2GB.
dbbishopAuthor Commented:
This is in regards to the field element of an xml format file for bulk inserts. The column importing into is varchar(max). The documentation states that the default if the MAX_LENGTH attribute isn't used is 8000 characters. What will happen if I use 1000000 to cover those few rows that may have a strong that is 30000 or 146841 characters?
Mark WillsTopic AdvisorCommented:
Pawan is correct. Use the special keyword 'MAX' instead of a number.

Then you dont have to worry too much about how many characters are being consumed in your VARCHAR(MAX) column

The VARCHAR() data type is variable in length, it will only use the space needed.

Or, are you asking about an old version of SQL - there used to be some limitations.? What version if I may ask ?

Or are you asking about a BCP fomat file used in bulk import ?
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

Mark WillsTopic AdvisorCommented:
Sorry about that,

I know see that you are in fact asking about a FORMAT FILE used for bulk operations and the MAX_LENGTH is one of the field attributes.

Totally distracted by the previous post... (long story)
Do you need to specify MAX_LENGTH ? You are mapping a field to a column attribute - right ?

Your column attributes are important e.g.   <COLUMN SOURCE="2" NAME="MyBigColumn" xsi:type="SQLNCHAR"/>

I think you'll find that MAX_LENGTH might not be as big a concern. Arguably better to use the field terminator.

What type of data are you loading ?

Any sample data ?
dbbishopAuthor Commented:
The problem is that the documentation states the default, if not specified, for VARCHAR data is 8000 characters and occasionally we will have data that exceeds that length. Will TERMINATOR 'trump' MAX_LENGTH or does bulk insert only load up to the MAX_LENGTH number of characters (either defined or implied value). I've done some work with bulk insert in the past but never with very long data strings. There are, in fact, field terminators (pipe character) in the data and they are being defined in the format file.

I could test, but I'd rather know the rules then rely on a test that might pass  under one condition and fail under another.
Mark WillsTopic AdvisorCommented:
Terminator is supposed to trump. FWIW I have only ever used terminator except for fixed format. The width (for import) is more a function of knowing / matching the receiving table / column

I think you will read somewhere in the fine print that the max_length is more for the export. In any case, it is only specifying the max length, so if you know that the raw data never exceeds 10000 characters, then use that as a safety net (but isnt needed) it doesnt reserve or pad out that length.

I very quickly knocked up a Table and used BCP to generate a format file with the following results:

Create table BCP_TABLE
 ID int identity,
 BigColumn varchar(max),
 DateColumn DateTime,
 XMLColumn xml,
 MoneyColumn money,
 DecColumn decimal(12,7),
 CharColumn char(12)

-- now build a format file

EXEC master..xp_cmdshell 'bcp EE.dbo.BCP_TABLE format nul -c -x -f C:\mrwtest\BCP_TABLE.xml -t, -T'
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="12"/>
  <FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," COLLATION="Latin1_General_CI_AS"/>
  <FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="24"/>
  <FIELD ID="4" xsi:type="CharTerm" TERMINATOR=","/>
  <FIELD ID="5" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30"/>
  <FIELD ID="6" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="41"/>
  <FIELD ID="7" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="12" COLLATION="Latin1_General_CI_AS"/>
  <COLUMN SOURCE="1" NAME="ID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="2" NAME="BigColumn" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="3" NAME="DateColumn" xsi:type="SQLDATETIME"/>
  <COLUMN SOURCE="4" NAME="XMLColumn" xsi:type="SQLNVARCHAR"/>
  <COLUMN SOURCE="5" NAME="MoneyColumn" xsi:type="SQLMONEY"/>
  <COLUMN SOURCE="6" NAME="DecColumn" xsi:type="SQLDECIMAL" PRECISION="12" SCALE="7"/>
  <COLUMN SOURCE="7" NAME="CharColumn" xsi:type="SQLCHAR"/>

Open in new window

You will note for the XMLColumn and the BigColumn, it doesnt have any notion of length but some of the others do.

So, just use the terminator....

But safety first. run the test. You will need a file that has a line with 10000 characters in it. can just import the one column.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
dbbishopAuthor Commented:
I am working on doing a test. We have some test data with fired widths in excess of 100K characters so that will be a good test.
Might not get to it until next week as it is late Friday and I'm off work until next Thursday so it may be awhile but I'll get back to you.

dbbishopAuthor Commented:
Works without MAX_LENGTH. Had some fields in excess of 65,000 characters and brought them in.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server

From novice to tech pro — start learning today.