SQL Server XML Format File

What is the best way to define MAX_LENGTH on a VARCHAR datatype that can exceed 8000 characters in an XML format file?
If you do not use the MAX_LENGTH attribute, MS states the default is 8000 characters. We have some cases (not many) where the length can be considerably more than that. I don't want to go sticking 1000000 in there when I don't know what the impact will be of importing tens of thousands of rows of data.
LVL 15
dbbishopAsked:
Who is Participating?
 
Mark WillsTopic AdvisorCommented:
Terminator is supposed to trump. FWIW I have only ever used terminator except for fixed format. The width (for import) is more a function of knowing / matching the receiving table / column

I think you will read somewhere in the fine print that the max_length is more for the export. In any case, it is only specifying the max length, so if you know that the raw data never exceeds 10000 characters, then use that as a safety net (but isnt needed) it doesnt reserve or pad out that length.

I very quickly knocked up a Table and used BCP to generate a format file with the following results:

Create table BCP_TABLE
(
 ID int identity,
 BigColumn varchar(max),
 DateColumn DateTime,
 XMLColumn xml,
 MoneyColumn money,
 DecColumn decimal(12,7),
 CharColumn char(12)
 )

-- now build a format file

EXEC master..xp_cmdshell 'bcp EE.dbo.BCP_TABLE format nul -c -x -f C:\mrwtest\BCP_TABLE.xml -t, -T'
/*
<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 <RECORD>
  <FIELD ID="1" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="12"/>
  <FIELD ID="2" xsi:type="CharTerm" TERMINATOR="," COLLATION="Latin1_General_CI_AS"/>
  <FIELD ID="3" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="24"/>
  <FIELD ID="4" xsi:type="CharTerm" TERMINATOR=","/>
  <FIELD ID="5" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="30"/>
  <FIELD ID="6" xsi:type="CharTerm" TERMINATOR="," MAX_LENGTH="41"/>
  <FIELD ID="7" xsi:type="CharTerm" TERMINATOR="\r\n" MAX_LENGTH="12" COLLATION="Latin1_General_CI_AS"/>
 </RECORD>
 <ROW>
  <COLUMN SOURCE="1" NAME="ID" xsi:type="SQLINT"/>
  <COLUMN SOURCE="2" NAME="BigColumn" xsi:type="SQLVARYCHAR"/>
  <COLUMN SOURCE="3" NAME="DateColumn" xsi:type="SQLDATETIME"/>
  <COLUMN SOURCE="4" NAME="XMLColumn" xsi:type="SQLNVARCHAR"/>
  <COLUMN SOURCE="5" NAME="MoneyColumn" xsi:type="SQLMONEY"/>
  <COLUMN SOURCE="6" NAME="DecColumn" xsi:type="SQLDECIMAL" PRECISION="12" SCALE="7"/>
  <COLUMN SOURCE="7" NAME="CharColumn" xsi:type="SQLCHAR"/>
 </ROW>
</BCPFORMAT>*/

Open in new window


You will note for the XMLColumn and the BigColumn, it doesnt have any notion of length but some of the others do.

So, just use the terminator....

But safety first. run the test. You will need a file that has a line with 10000 characters in it. can just import the one column.
0
 
Pawan KumarDatabase ExpertCommented:
Please use VARCHAR(MAX)

it can store up to 2GB of data.

From Microsoft.

varchar [ ( n | max ) ]
Variable-length, non-Unicode string data. n defines the string length and can be a value from 1 through 8,000. max indicates that the maximum storage size is 2^31-1 bytes (2 GB). The storage size is the actual length of the data entered + 2 bytes. The ISO synonyms for varchar are char varying or character varying.

https://msdn.microsoft.com/pl-pl/library/ms176089(v=sql.110).aspx

if we provide size , we can go upto 8000 characters
else if we use Max then we can go upto 2GB.
0
 
dbbishopAuthor Commented:
This is in regards to the field element of an xml format file for bulk inserts. The column importing into is varchar(max). The documentation states that the default if the MAX_LENGTH attribute isn't used is 8000 characters. What will happen if I use 1000000 to cover those few rows that may have a strong that is 30000 or 146841 characters?
0
Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
Mark WillsTopic AdvisorCommented:
Pawan is correct. Use the special keyword 'MAX' instead of a number.

Then you dont have to worry too much about how many characters are being consumed in your VARCHAR(MAX) column

The VARCHAR() data type is variable in length, it will only use the space needed.

Or, are you asking about an old version of SQL - there used to be some limitations.? What version if I may ask ?

Or are you asking about a BCP fomat file used in bulk import ?
0
 
Mark WillsTopic AdvisorCommented:
Sorry about that,

I know see that you are in fact asking about a FORMAT FILE used for bulk operations and the MAX_LENGTH is one of the field attributes.

Totally distracted by the previous post... (long story)
 
Do you need to specify MAX_LENGTH ? You are mapping a field to a column attribute - right ?

Your column attributes are important e.g.   <COLUMN SOURCE="2" NAME="MyBigColumn" xsi:type="SQLNCHAR"/>

I think you'll find that MAX_LENGTH might not be as big a concern. Arguably better to use the field terminator.

What type of data are you loading ?

Any sample data ?
0
 
dbbishopAuthor Commented:
The problem is that the documentation states the default, if not specified, for VARCHAR data is 8000 characters and occasionally we will have data that exceeds that length. Will TERMINATOR 'trump' MAX_LENGTH or does bulk insert only load up to the MAX_LENGTH number of characters (either defined or implied value). I've done some work with bulk insert in the past but never with very long data strings. There are, in fact, field terminators (pipe character) in the data and they are being defined in the format file.

I could test, but I'd rather know the rules then rely on a test that might pass  under one condition and fail under another.
0
 
dbbishopAuthor Commented:
I am working on doing a test. We have some test data with fired widths in excess of 100K characters so that will be a good test.
Might not get to it until next week as it is late Friday and I'm off work until next Thursday so it may be awhile but I'll get back to you.

Thanks!
0
 
dbbishopAuthor Commented:
Works without MAX_LENGTH. Had some fields in excess of 65,000 characters and brought them in.
1
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.