Avatar of Edgar Cole
Edgar Cole
Flag for United States of America asked on

rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6]

I'm using rsync to copy the home directories on an AIX node with close to 12,000 accounts to a remote AIX node, when I get the following error message:

sending incremental file list
home/
home/102544/
home/102544/.profile
ERROR: out of memory in flist_expand [sender]
rsync error: error allocating core memory buffers (code 22) at util.c(117) [sender=3.0.6]


I'm guessing it's because there are so many directories on the source node. Is there an option in rsync that I can use to handle this issue? Otherwise, I'll have to devise a way to divide the directory list into smaller chunks.
Storage SoftwareUnix OSOperating Systems

Avatar of undefined
Last Comment
Edgar Cole

8/22/2022 - Mon
ASKER CERTIFIED SOLUTION
woolmilkporc

Log in or sign up to see answer
Become an EE member today7-DAY FREE TRIAL
Members can start a 7-Day Free trial then enjoy unlimited access to the platform
Sign up - Free for 7 days
or
Learn why we charge membership fees
We get it - no one likes a content blocker. Take one extra minute and find out why we block content.
Not exactly the question you had in mind?
Sign up for an EE membership and get your own personalized solution. With an EE membership, you can ask unlimited troubleshooting, research, or opinion questions.
ask a question
woolmilkporc

Another thing: What is your rsync version?

If it's still 3.03 your issue could be due to a bug which has been corrected in later releases.

http://www.perzl.org has 3.08-1 and I'd strongly suggest upgrading to this version should your rsync really be so old.

Edgar Cole

ASKER
This is what was returned by the rpm command:

rsync-3.0.6-1

Unfortunately, I'm not running the 64-bit version, but I am at version 3.0.6.

Just to be sure, the variable is named LDR_CNTRL=MAXDATA... not LDR_CNTRL_MAXDATA?
woolmilkporc

Yes, it's

LDR_CNTRL=MAXDATA=...
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Edgar Cole

ASKER
Thanx. I'll let you know how it goes.
Edgar Cole

ASKER
Phooey!

sending incremental file list
/home
home/109257/.sh_history
home/109257/smit.log
home/109257/smit.script
home/109257/smit.transaction
home/109459/.profile
home/109565/.profile
home/109995/.sh_history
rsync: connection unexpectedly closed (10479 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]


The good news is that it's no longer complaining about memory. The bad news is that I have no idea what's wrong. The other rsync transfers I run from this node are working fine. I'm hoping that rsync has a trace option that I can activate to generate more diagnostics.  
woolmilkporc

Could it be that the destination disk at the receiving end is full?

How about the ulimit setting of the target userid?

Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
woolmilkporc

Add this to your rsync command to enable logging:

--log-file=/tmp/rsync_send.log

And to make the remote side log what's going on add this to your rsync command:

--rsync-path="rsync --log-file=/tmp/rsync_receive.log"
Edgar Cole

ASKER
Destination looks good and the ulimit says unlimited.
woolmilkporc

Is rsync installed at all at the remote end? (Sorry for asking!)
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
Edgar Cole

ASKER
Yep, it is. :-)

I've got a couple of things I'm going to try before installing 3.0.8 or a 64-bit version. I'll let you know what happens.
Edgar Cole

ASKER
I created a TAR of the source directory... almost 1 GIGABYTE!! :-O
woolmilkporc

The size alone should be no problem, as opposed to an exorbitant number of files.

What do you get with

tar -tvf [tarfile] | wc -l

?

wmp
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
Edgar Cole

ASKER
The answer to your question is 28,874.

However, I'm really confused now! A cursory examination suggests that at least the top level directories are being created. If I take a count in the home directory on the source and compare that to a count in the home directory on the destination, I get the same number; 11,830. The total number of subdirectories in the home directory on the source is 12167, but only the top-level directories (i.e., /home/username) are making it to the destination (yes, the recursive flag is set).

The latest error message looks like this:

2011/08/11 09:42:33 [381248] rsync: connection unexpectedly closed (12228 bytes received so far) [sender]
2011/08/11 09:42:33 [381248] _exit_cleanup(code=12, file=io.c, line=600): entered
2011/08/11 09:42:33 [381248] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
2011/08/11 09:42:33 [381248] _exit_cleanup(code=12, file=io.c, line=600): about to call exit(12)

It's  running longer and appears to be getting farther, but that's about as much as I know so far. I did set verbosity at four (-vvvv) and captured the output in a log. I'm not seeing anything in the log that's conspicuous. It seems to be running along just fine when it suddenly bails out. Would fiddling with the data segment size some more do me any good? I've got a couple more things to try before I install 3.0.8.
woolmilkporc

Did you set up the remote logging as in my comment #36346851 (maybe with increased verbosity)?

--rsync-path="rsync -vvvv --log-file=/tmp/rsync_receive.log"

What do you find in /tmp/rsync_receive.log on the remote machine?

wmp
Edgar Cole

ASKER
I cut the list in half and restarted the transfer. It's been running for more than an hour and seems to have stalled. I'm going to cancel it and set up the remote logging as you suggested. I'll let you know what happens.
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
woolmilkporc

Are there any messages in the remote errpt which could be related to this issue?

wmp
Edgar Cole

ASKER
I'm getting this message when I try installing version 3.0.8 of rsync:

The following errors occurred during installation:
error: failed dependencies:
        popt is needed by rsync-3.0.8-1
        libpopt.a(libpopt.so.0) is needed by rsync-3.0.8-1


I get the same thing when I try installing the 64-bit version of 3.0.6.
Edgar Cole

ASKER
...and now popt needs gettext. When will it end?
Get an unlimited membership to EE for less than $4 a week.
Unlimited question asking, solutions, articles and more.
woolmilkporc

OK, why not install popt and popt-devel?

http://www.perzl.org/aix/index.php?n=Main.Popt

Install both using the "--nodeps" option. This will bypass the requirement for gettext. Should you have gettext installed - no need for that option, though.

wmp
Edgar Cole

ASKER
In the interest of time I had to find a solution that, while not as elegant, was equally effective. The result was a script that calls the split command to divide the file list into smaller chunks. It looks something like this...

find /home/* -prune | sort | split -l 500 -a 2 - filesync

This command pipe will divide the output generated by the find command into individual files of no more than 500 lines each. In a subsequent step (not shown here), those files are fed sequentially to the rsync command.
Edgar Cole

ASKER
I was not able to try everything that was suggested, so it's difficult to evaluate this solution. Specifically, I was not able to upgrade the version of rsync that I have installed. I did use the LDR_CNTRL variable, which seemed to help. I still had to whittle down the input list from 12,000 to 500.
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23