# Mac OSX Spotlight indexing takes forever

Hi,

I have a user running OSX 10.6.4 with a remote home directory on my windows server.  Every time she logs into her machine, the spotlight search will reindex her files and it takes about 5 hours which means she cannot use it for most of the day.  Her user profile is very large, about 40gb.  Is it just the sheer size of her profile that makes it take so long?  I've tried excluding her 20gb of music from the spotlight search, but there is still about 10gb of email and 10gb of files that take forever to index.  Any ideas?

###### Who is Participating?

x

Commented:
Forcing local home directories has advantages, as was discussed previously. You had indicated that it was forbidden, however.

Did you ever do this:
Can you use \\servername.xxx.edu\macprofiles$\tony3 and see if you get the same errors? I was wondering about DNS because of this error: "mdworker(320) deny network-outbound /private/var/run/mDNSResponder" which indicates that mdworker was trying to use bonjour (aka mDNSResponder) It would be good to ensure that the errors are not a red herring. Also, I am assuming that your test machine is also having the same problems with a lengthy period of poor performance while spotlight appears to be indexing. Please advise if this assumption is wrong. 0 Commented: Since OS X is not in control of the remote file share, it has to look at every single file to look for changes. If you need spotlight search enabled for the remote files, you might be better using synchronization rather than connecting remotely. (Like with Windows Live Sync, http://sync.live.com). This way, OS X is in control of the files and Spotlight doesn't have to look at every single file during the index. Otherwise, just take the home directory out of the spotlight index. 0 Author Commented: Is there any chance that this can be caused by corrupt files. I read that if your logs show errors for the mdworker process, they recommend to delete the offending files. My log indeed shows a ton of errors like this... mdworker[2989] Invalid char for PropertyName in line 18 I have no clue how to find the file it is referring to in line 18. 0 Commented: It sounds like the spotlight index is corrupt. I suggest you delete it and let OS X rebuild it. See: http://www.macintouch.com/tigerreview/spotlight.html#fixes 0 Commented: 0 Commented: 0 Commented: 0 Commented: Actually, I am surprised it is indexing a remote Windows volume. I had understood that Spotlight worked only on Mac volumes. 0 Commented: Let me preface the following by a caution that I know nothing about AD. With that out of the way: 1. If this user is always on the same machine (you said this is the only mac on the network), it might be best to set up a local account on her machine. Pretty sure that spotlight will, by default, index network homes (but not other remote volumes unless you use coercion and trickery), Even with network homes under under OS X server — which has server side spotlight indexing — there are bottlenecks created by caches being written to and read from a remote server. There are ways to redirect the cache files to the local disk, but I understand that it can be an adventure if you are running Office Mac applications. Setting up the user as a mobile account (I know nothing about AD and am not sure if it can be done in your setting, though) would be better. A further alternative (can be done with OD and, I suspect also with AD) would be to configure the network user to have a home folder at /Users/ This enables the network user to have a home directory in the same place as would be the case for local account - on the local hard drive in the /Users/ folder. To do this with OS X server, after configuring the network user with Workgroup Manager, you copy the home directory into the local drive in a way that maintains the privileges (network user is owner, etc), then log in as before with the network username and password. 0 Author Commented: I've tried rebuilding the index and it still re-indexes after each login. In response to nxnw's suggestions... Security audits mandate that these home directories must not be stored on the local machine so I can't go with local profiles. At the moment i'm experimenting with nhr.dmg to redirect the local caches. I'll let you guys know if it helps. 0 Commented: OK, some other thoughts, then: Out of an abundance of caution, how did you go about rebuilding the suspect index? I'm not sure the usual utilities will get the index for a network home, because it is kept in a different place (/var/db/spotlight/ I think). Also, have you tried making the network home private (in system preferences/spotlight) as a temporary workaround to stop the endless indexing? Finally, troublemaker that I am: If a local home directory, that could be encrypted, violates security audit standards, how about unencrypted local cache files and spotlight indexes? 0 Commented: Other possibilities: - a corrupt file that spotlight is trying to index (would likely show up in system logs) - a privilege issue with the directory containing the index 0 Author Commented: The user and creator/owner have full control over the .spotlight-v100 dir so I dont think it's that unless there is some other group/user i'm missing. How would I go about finding the corrupt file that the index is choking on. All it tells me is .. mdworker[2989] Invalid char for PropertyName in line 18 without any real file name or location. Is there a way I can track it down? I've ran repair permissions and repair disk without any luck. 0 Commented: 0 Commented: 0 Author Commented: Responding to your earlier post. 1.) I rebuilt the index by logging in as the user and running sudo mdutil -E / I'm getting permission denied when trying to browse the /var/db locations, will try in single user mode 2.) Since the users are adamant about using the spotlight, i've just had them use a password protected screen saver instead of logging off 3.) I do realize that one could get some info from the local cache files. I would prefer an encrypted local profile and remote file shares for ease of backup but i'm just following orders whether they make sense or not. :) 0 Commented: If it turns out the problem is the .ds store files, here is how to stop the Mac from creating them on the network volume: http://hints.macworld.com/article.php?story=2005070300463515 0 Commented: "The user and creator/owner have full control over the .spotlight-v100 dir " That isn't where the index is. It is in /var/db/ You can get at it in terminal by opening a session as a superuser with the command: sudo bash If rebuilding the index doesn't fix it, the reference provided by Strung in 33692925 looks really good. 0 Author Commented: I've tried rebuilding that index under /var/db and it doesn't help. The network accounts are still reindexing the entire profile each time they login. The full command lsof | grep mdimport | grep /Users/whoami doesnt do anything for me, but "lsof | grep mdimport " reports the following mdworker 229 tony3 txt REG 14,2 51136 1168897 /System/Library/Spotlight/Application.mdimporter/Contents/MacOS/Application mdworker 229 tony3 txt REG 14,2 84352 778011 /System/Library/Spotlight/Image.mdimporter/Contents/MacOS/Image mdworker 229 tony3 txt REG 14,2 61632 830265 /System/Library/Spotlight/iCal.mdimporter/Contents/MacOS/iCal mdworker 229 tony3 txt REG 14,2 60320 1138787 /System/Library/Spotlight/Audio.mdimporter/Contents/MacOS/Audio mdworker 229 tony3 txt REG 14,2 50560 777637 /System/Library/Spotlight/Archives.mdimporter/Contents/MacOS/Archives mdworker 229 tony3 txt REG 14,2 51696 778020 /System/Library/Spotlight/PDF.mdimporter/Contents/MacOS/PDF mdworker 229 tony3 txt REG 14,2 145072 1133194 /System/Library/Spotlight/RichText.mdimporter/Contents/MacOS/RichText mdworker 229 tony3 txt REG 14,2 95552 1133774 /System/Library/Spotlight/vCard.mdimporter/Contents/MacOS/vCard mdworker 229 tony3 txt REG 14,2 77200 1146905 /System/Library/Spotlight/Mail.mdimporter/Contents/MacOS/Mail mdworker3 234 tony3 txt REG 14,2 2894560 790256 /Library/Spotlight/Microsoft Office.mdimporter/Contents/MacOS/Microsoft Office mdworker3 234 tony3 txt REG 14,2 2360240 790247 /Library/Spotlight/Microsoft Office.mdimporter/Contents/Frameworks/MetroFramework.framework/Versions/12/MetroFramework mdworker3 234 tony3 9r REG 14,2 30450 790258 /Library/Spotlight/Microsoft Office.mdimporter/Contents/Resources/Microsoft Office.rsrc The logs create right after login and after the rebuilding begins are as follows 9/17/10 8:38:33 AM mds[29] (Normal) DiskStore: Rebuilding index for /Network/Servers/xyz.edu/macprofiles$/tony3
9/17/10 8:38:33 AM      mds[29]      (Normal) DiskStore: Creating index for /Network/Servers/xyz.edu/macprofiles$/tony3 9/17/10 8:39:35 AM mdworker32[321] kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged. 9/17/10 8:39:41 AM mdworker[320] socket(PF_ROUTE) failed: Operation not permitted 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny system-socket 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny network-outbound /private/var/run/mDNSResponder 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny network-outbound /private/var/run/mDNSResponder 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny network-outbound /private/var/run/mDNSResponder 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny network-outbound /private/var/run/mDNSResponder 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny network-outbound /private/var/run/mDNSResponder 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny network-outbound /private/var/run/mDNSResponder 9/17/10 8:39:41 AM mdworker[320] socket(PF_ROUTE) failed: Operation not permitted 9/17/10 8:39:41 AM sandboxd[324] mdworker(320) deny system-socket 0 Commented: The first group from the lsof command are all local. Not relevant. The second group, I'm pretty sure, means that mdworker is trying to index the network home directory, but the internal firewall is blocking it. If that is correct, the user may be able to use spotlicht, but his searches will not find documents added to his home directory since the problem started. I'm not sure if turning off the internal firewall (system preferences/security) will completely turn off sandboxd, but it's worth exploring (on the path to a solution) 0 Author Commented: The firewall was already turned off. Not sure where to go from here. 0 Commented: I think sandboxd does more than just the firewall, and the logs do indicate that it is blocking network access to spotlight indexing. Three shots in the dark, - run the 10.6.4 combo installer, in case it is being caused by system corruption of some kind; - deep clean your caches, in case it is being caused by corrupt caches; - turn off Time Machine. 0 Author Commented: Time machine was off, the Onyx cache cleaning didnt help, and it can't be system corruption because its happening on 13 different machines. I'm waiting on a response from some of the Apple and MS forums. 0 Commented: Ahh. I was thrown off by "I have a user running OSX 10.6.4 with a remote home directory" and have been assuming it was a single machine. Almost all of the suggestions provided above would be very unlikely in a scenario of 13 different machines. The fact that this was happening on multiple machines was a critical piece of information. In the circumstances, please provide more info: • [b]can you confirm that recent changes are not being indexed[b] (which would be the case if I am interpreting the log entry correctly)? • [b]Are all of the affected macs running 10.6.4? 10.6.x?[b] • [b]Are the 13 affected machines all of your macs?[b] I suspect that your problem is due to a bug in the OS. The above may confirm this and possibly suggest a workaround. If it is a bug, it should be reported to Apple. Finally, if the actual problem is — in fact — that spotlight is failing to index OS X network homes on a windows server, your problem should be described accordingly. A more accurate description will help get you a solution. 0 Author Commented: Sorry for not being more descriptive, after entering this problem on several forums things get a little blurry :) I'm pretty sure the recent changes are actually being indexed, but i'll verify. My test account is only 3gb or so and I let it completely index and the spotlight was able to find an email that came in that day. Give me 30 min and i'll try this again. All the affected macs are running 10.6.4, and the 13 macs are my entire test population at this point (If I can get this stupid thing fixed, i've got 10 more to add) 0 Author Commented: I have confirmed that recent changes are being indexed by spotlight. I'm able to find text from emails sent this morning after logging in and waiting around 45 min to index everything. The issue that remains is why it insists on re-indexing everything every time time the user logs in. 0 Commented: And that is the same machine that is getting these errors in the log? mdworker(320) deny network-outbound /private/var/run/mDNSResponder mdworker[320] socket(PF_ROUTE) failed: Operation not permitted sandboxd[324] mdworker(320) deny system-socket By the way, how does the mac know where to look for the network homes? Do you login to the network homes using a FQDN? Are the correct local DNS servers shown in network settings? 0 Author Commented: The macbook back in my office gave me those errors. The clients in a different building were similar. I can provide some of their specific log errors tomorrow. In Active Directory, the user account has a tab for "profiles". You can specify the home folder to connect to a network volume. My test account uses... \\servername\macprofiles$\tony3

The actual users have...
\\servername.xxx.edu\macprofiles$\tony3 The problem exists no matter if I use the FQDN or not. 0 Commented: Can you use \\servername.xxx.edu\macprofiles$\tony3 and see if you get the same errors? I was wondering about DNS because of this error:
"mdworker(320) deny network-outbound /private/var/run/mDNSResponder"
which indicates that mdworker was trying to use bonjour (aka mDNSResponder)

It would be good to ensure that the errors are not a red herring.

Also, I am assuming that your test machine is also having the same problems with a lengthy period of poor performance while spotlight appears to be indexing. Please advise if this assumption is wrong.
0

Author Commented:
I've had good luck with using a brand new profile and not importing the old one, only the data itself.  I'll give it a few more days to test it before posting the results.

The indexing seems to go very quickly.
0

Commented:
So what was different?
0

Author Commented:
Never was able to resolve this issue.  We went back to forcing local home directories and just passing the authentication through AD without the remote home directories.  I guess it wasnt meant to be.
0

Author Commented:
Ya, we tried using IP as well as fqdn with no change.  My test machine was having the same lengthy indexing process as the user machines with a new profile as well as an imported one.  After enough complaints from the users we got the OK from their security watchdogs to go back to the local profiles.

I've since left this job so I guess the issue is closed :)
0

Author Commented:
tried everything, thanks for the help
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.