rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6]

Another thing: What is your rsync version?

If it's still 3.03 your issue could be due to a bug which has been corrected in later releases.

http://www.perzl.org has 3.08-1 and I'd strongly suggest upgrading to this version should your rsync really be so old.

This is what was returned by the rpm command:

rsync-3.0.6-1

Unfortunately, I'm not running the 64-bit version, but I am at version 3.0.6.

Just to be sure, the variable is named LDR_CNTRL=MAXDATA... not LDR_CNTRL_MAXDATA?

Yes, it's

LDR_CNTRL=MAXDATA=...

Thanx. I'll let you know how it goes.

Phooey!

sending incremental file list
/home
home/109257/.sh_history
home/109257/smit.log
home/109257/smit.script
home/109257/smit.transaction
home/109459/.profile
home/109565/.profile
home/109995/.sh_history
rsync: connection unexpectedly closed (10479 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

The good news is that it's no longer complaining about memory. The bad news is that I have no idea what's wrong. The other rsync transfers I run from this node are working fine. I'm hoping that rsync has a trace option that I can activate to generate more diagnostics.  

Could it be that the destination disk at the receiving end is full?

How about the ulimit setting of the target userid?

Add this to your rsync command to enable logging:

--log-file=/tmp/rsync_send.log

And to make the remote side log what's going on add this to your rsync command:

--rsync-path="rsync --log-file=/tmp/rsync_receive.log"

Destination looks good and the ulimit says unlimited.

Is rsync installed at all at the remote end? (Sorry for asking!)

Yep, it is. :-)

I've got a couple of things I'm going to try before installing 3.0.8 or a 64-bit version. I'll let you know what happens.

I created a TAR of the source directory... almost 1 GIGABYTE!! :-O

The size alone should be no problem, as opposed to an exorbitant number of files.

What do you get with

tar -tvf [tarfile] | wc -l

?

wmp

The answer to your question is 28,874.

However, I'm really confused now! A cursory examination suggests that at least the top level directories are being created. If I take a count in the home directory on the source and compare that to a count in the home directory on the destination, I get the same number; 11,830. The total number of subdirectories in the home directory on the source is 12167, but only the top-level directories (i.e., /home/username) are making it to the destination (yes, the recursive flag is set).

The latest error message looks like this:

2011/08/11 09:42:33 [381248] rsync: connection unexpectedly closed (12228 bytes received so far) [sender]
2011/08/11 09:42:33 [381248] _exit_cleanup(code=12, file=io.c, line=600): entered
2011/08/11 09:42:33 [381248] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
2011/08/11 09:42:33 [381248] _exit_cleanup(code=12, file=io.c, line=600): about to call exit(12)

It's  running longer and appears to be getting farther, but that's about as much as I know so far. I did set verbosity at four (-vvvv) and captured the output in a log. I'm not seeing anything in the log that's conspicuous. It seems to be running along just fine when it suddenly bails out. Would fiddling with the data segment size some more do me any good? I've got a couple more things to try before I install 3.0.8.

Did you set up the remote logging as in my comment #36346851 (maybe with increased verbosity)?

--rsync-path="rsync -vvvv --log-file=/tmp/rsync_receive.log"

What do you find in /tmp/rsync_receive.log on the remote machine?

wmp

I cut the list in half and restarted the transfer. It's been running for more than an hour and seems to have stalled. I'm going to cancel it and set up the remote logging as you suggested. I'll let you know what happens.

Are there any messages in the remote errpt which could be related to this issue?

wmp

I'm getting this message when I try installing version 3.0.8 of rsync:

The following errors occurred during installation:
error: failed dependencies:
        popt is needed by rsync-3.0.8-1
        libpopt.a(libpopt.so.0) is needed by rsync-3.0.8-1

I get the same thing when I try installing the 64-bit version of 3.0.6.

...and now popt needs gettext. When will it end?

OK, why not install popt and popt-devel?

http://www.perzl.org/aix/index.php?n=Main.Popt

Install both using the "--nodeps" option. This will bypass the requirement for gettext. Should you have gettext installed - no need for that option, though.

wmp

In the interest of time I had to find a solution that, while not as elegant, was equally effective. The result was a script that calls the split command to divide the file list into smaller chunks. It looks something like this...

find /home/* -prune | sort | split -l 500 -a 2 - filesync

This command pipe will divide the output generated by the find command into individual files of no more than 500 lines each. In a subsequent step (not shown here), those files are fed sequentially to the rsync command.

I was not able to try everything that was suggested, so it's difficult to evaluate this solution. Specifically, I was not able to upgrade the version of rsync that I have installed. I did use the LDR_CNTRL variable, which seemed to help. I still had to whittle down the input list from 12,000 to 500.