what is maximum memory for dell xps m1730?

Maximum appears to be 8GB of RAM.

http://shop.crucial.com/upgrade/compatible-memory-for/Dell/XPS+M1730/list.html

It has to be 4 gigs on each slot in order for the bios to see it as DDR2 PC2-6400? and the OS has to be 64 bits?

Yes, and yes (for full use of the RAM accordingly).

It is best to keep both sticks the same size and speed (speed especially). If the Operating System is only 32 Bit, then it will only see 4 GB of RAM MAX so there is no need in buying more than one 4GB stick or two 2GB sticks. If you have a 64 Bit OS then you can go up to two 4GB sticks on memory of either PC2-5300 or PC2-6400. It doesn't HAVE to be two 4GB sticks. You can use a 4GB stick and a 1GB stick, or other combinations but try to keep the speed the same with either 5300 OR 6400 for both sticks.

Yes and not entirely true actually.

If you are running Windows 7 you need the 64 bit version of Windows 7 Home Basic or higher to see the memory.  The Windows 7 Starter edition does not see 8 GB even with a 64 bit.

thanks you guys

Here it is all you want to know:

About replacing/adding memory modules: http://support.dell.com/support/edocs/systems/xpsm1730/en/sm/memory.htm#wp1180190

Owner's manual, where you can find info, about memory supported in your laptop maximum: http://support.dell.com/support/edocs/systems/xpsm1730/en/om_en/pdf/TM373A04MR.pdf

It says maximum memory is 4 GB (via 2 x 2 GB SODIMM DDR-II modules), which is info from some years ago, but I assume that if you UPDATE your CMOS/BIOS firmware to latest version, your laptop should support also 2 x 4 GB DDR-II modules, suming up to 8 GB of total memory.