Troubleshooting File Sharing Issues with With Mac OS X and Windows

Connectivity using SMB

Symptoms

Domain controllers by default are enabled to “Digitally Sign Communications” always for network clients and servers (prior to a server being promoted to a DC this setting will be disabled. 2008 R2 only enables this for servers). When this is the case a Mac running Tiger (10.4.x) or below will not be able to authenticate to a Windows server. After entering your credentials you’ll receive an error saying something along the lines of “server could not be found,” when clearly it’s there because it asked you to authenticate. All the hassle is actually an easy fix on both 2003 and 2008. This disconnect is attributed to SMB signing.

Alternately, in Mac OS 10.5 or 10.6 you may have issues saving files to the server after you've authenticated (some of these issues may have been or will be resolved by Apple OS Software Updates). The symptoms are different in 10.5 and 10.6 where 10.6 can actually copy small files to the server using the Finder but when trying to copy larger files the Finder acts like the copy finishes then leaves a 0kb file on the server. Mac OS 10.5 may actually authenticate and list files but have read-only access to the server.

Another known issue that occurs after authentication is is not being able to see all of your folders, or none of your folders, when connected to a Windows share. This also causes 10.6 machines to not be able to copy larger files to the server. Alternately, you may have very slow logins. This has something to do with the port usage in Samba on the Mac OS X side and may or may not be resolved by upcoming patches in 10.5 and 10.6. There's a simple workaround but can cause problems to those of you who browse to your servers or shares using the network browser in the Finder. See the "Adding the SMB Port to Connection String" for 10.5 and 10.6 below. Similarly, if you're not seeing all of your file shares when browsing a share on a Windows XP computer you may not be able to see all the files and folders (particularly within the Documents and Settings folder). In this case you may have "simple file sharing" enabled preventing enabling of Full Control for Everyone within the share permissions (not the security permissions).

With Windows Server 2008 new security lock downs have produced new connectivity/authentication issues with Mac and some Windows machines. 2008 Servers ship with the default security setting of "Send NTLMv2 response only" (2008 R2 comes with this setting "not defined" before and after promotion to a DC). Macs (prior to 10.58 "Leopard") and some earlier versions of Windows do not support this. Do the following to lighten the server's security settings. I've seen only sporadic issues with this and should be resolved for 10.5 and 10.6 as I've just tested with NTLMv2 only enabled.

Resolutions

Windows 2003/2008/Vista/7 Adjustments - Disabling SMB Signing

Microsoft network client: Digitally sign communications (always) set to disabled
Microsoft network server: Digitally sign communications (always) set to disabled


Windows XP - If you're unable to view certain files and folders when connecting to a Windows XP share from Mac OS X make sure "Simple File Sharing" is not enabled and you have given the Everyone user Full Access under the share permissions:


*Note: Use the security tab to limit permissions to specific users

Mac OS X 10.5/10.6 - Adding the SMB Port to Connection String


Windows 2008/Vista/7 Adjustments - By default Windows 2008 Server (not R2) sends only NTLMv2 responses. This prevents authentication from Macs and non-updated versions of Windows XP.


Windows 2008 Cluster Servers


From: Macintosh 10.x client cannot connect to File Server on Windows 2008 Failover Cluster

In Win2008 Failover Clustering we introduced a new Share Scoping feature, where shares are only available under the clustered name instance they are associated with.  In order to make the connection you need to connect to the share name associated with the cluster file share server name.

With MAC OS 10.4 we've seen it work correctly with Win2008 Clustered file shares.
With MAC OS 10.5 there appears to be a change to the client where it only tries the IP Address, and never tries the NetBIOS name or FQDN.  And in turn does not work with Win2008 Clustered file shares.

This same issue is apparent in Mac OS 10.6 with recent experience and none of our numerous attempted workarounds were sufficient. After contacting Microsoft support on the issue they confirmed it would not work due to the quoted issue above. The workaround and recommended solution is to use Group Logic's ExtremeZIP.

Connectivity Using AFP

Symptoms

In Windows 2003 it’s easy enough to install Apple File Sharing from Add/Remove Programs and create Macintosh file shares in addition to your Windows shares. After you have Appletalk installed when you go to create a share Within Computer Management you’ll be given the option to create Apple shares and Windows shares. Once you’ve created your Mac share you may find that when you go to mount your share you receive the same authentication error as you did when trying to mount the share over SMB. There’s an easy fix for this as well.

Resolutions

AFP/Windows Authentication


Now, that you're able to authenticate to the server over AFP but now you're stuck with read-only permissions no matter what you do. Also, another simple fix.

AFP/Windows Read-Only


AFP/Windows Character and File Name Limitations

There's a couple of limitations when using AFP. Some characters that are allowed in file name on the Mac aren't allowed in file names in Windows. The following characters are not allowed in Windows file names and there’s a 31 character file name length limitation.

< > : " / \ | ? *

Select all Open in new window

Additional Tips

If you're having trouble copying large files or any files to a Windows share you can often copy the files using the Terminal with RSYNC or the CP command. The issues in Snow Leopard not being able to copy large files has something to do with the Finder and not the actual connection.

If you're having trouble seeing files and folders even after the suggested changes, or you're having trouble with files and folders on the Windows machines after Macs have began using Windows file shares, you should check your shared files/folders for leading or trailing spaces and/or unsupported characters in the file names.

Closing

With the current exception of SMB file sharing in Windows 2008 Failover Clusters file sharing between Mac OS X and Windows can be accomplished with little problems. Although, if you can afford it, I recommend using ExtremeZIP or a Mac OS X server to reduce issues caused by sharing files over SMB or Windows 2003's AFP to prevent small unavoidable cross platform issues. If you need to backup your files from the Windows side ExtremeZIP is a great alternative to introducing a Mac server in your environment and has worked flawlessly for me in all versions of Windows server.

If you're concerned about security in disabling these functions you should read the article below named "So what is SMB Signing all about?"

Links and Sources

Apple Discussion - Topic : Mac OS X 10.4: Error -36 alert displays when connecting to a Windows server
MacWindows - Snow Leopard File Sharing Issues and Reports
MacWindows - Mac OS X 10.5 Leopard Cross-platform Issues
TechNet Social - Macintosh 10.x client cannot connect to File Server on Windows 2008 Failover Cluster
Apple Support - Mac OS X: How to connect to Windows File Sharing (SMB)
MSMVPS KWSupport - So what is SMB Signing all about?
TechNet - Network security: LAN Manager authentication level
TechNet - Microsoft network client: Digitally sign communications (always)
Tech Republic - Allow Windows Vista, Server 2008 systems to interact with older Samba installations
Mac OS X Hints - Fix a Vista to Mac failure to connect problem Network
Group Logic ExtremeZIP
Disable Simple File Sharing in XP