Solved

RSYNC Errors 30 and 5

Posted on 2011-09-27
24
2,624 Views
Last Modified: 2013-12-01
I am having some major issues with rsync. The IT person before me set this up, I am not really familiar with how this works. I do know that the machine that is taking the backups is an Ubuntu Linux server. One of the drives it is trying to backup is on a 2003 Windows Server. Here are some snippets from the rsync log file.

Sep 26 14:10:48 brown rsyncd[888]: inflate returned -3 (679 bytes)
Sep 26 14:10:48 brown rsyncd[888]: rsync error: error in rsync protocol data stream (code 12) at token.c(546) [receiver=3.0.7]
Sep 26 14:10:48 brown rsyncd[888]: rsync: connection unexpectedly closed (79577 bytes received so far) [generator]
Sep 26 14:10:48 brown rsyncd[888]: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [generator=3.0.7]
Sep 26 14:10:51 brown rsyncd[932]: Unknown Parameter encountered: "patht"
Sep 26 14:10:51 brown rsyncd[932]: IGNORING unknown parameter "patht"
Sep 26 14:10:51 brown rsyncd[932]: name lookup failed for 10.0.0.22: Name or service not known
Sep 26 14:10:51 brown rsyncd[932]: connect from UNKNOWN (10.0.0.22)
Sep 26 14:10:51 brown rsyncd[932]: rsync to redMon/Shares/ from UNKNOWN (10.0.0.22)
Sep 26 14:10:51 brown rsyncd[932]: receiving file list
Sep 26 14:10:52 brown rsyncd[932]: rsync: readlink_stat("/Shares/Temp" (in redMon)) failed: Input/output error (5)
Sep 26 14:10:52 brown rsyncd[932]: IO error encountered -- skipping file deletion
Sep 26 14:10:52 brown rsyncd[932]: rsync: failed to set times on "/Shares/." (in redMon): Read-only file system (30)
Sep 26 14:10:52 brown rsyncd[932]: rsync: failed to set times on "/Shares/ACH" (in redMon): Read-only file system (30)
Sep 26 14:10:52 brown rsyncd[932]: rsync: recv_generator: failed to stat "/Shares/ACH/ALAMON0074" (in redMon): Input/output error (5)

This one is popping up repeatedly in the rsync log file

Sep 26 14:10:53 brown rsyncd[932]: rsync: mkstemp "/Shares/.0405.PVKnPM" (in redMon) failed: Read-only file system (30)

Any help would be greatly appreciated!!
0
Comment
Question by:jparker81
  • 10
  • 9
  • 5
24 Comments
 
LVL 39

Expert Comment

by:noci
ID: 36714972
Did anything happen to the /Shares filesystem?
is it still in tact?

Appearantly the filesystem is write protected, that can be fixed question is how did it become Read/Only
there is a mechanism that allows you to keep a disk online but  read-only after errors.
In that case you need to verify the filesystem first.

Please do an readonly check of the file system.
for ext2,3,4:    fsck.extX -n /dev/...         # use 2,3 or 4 on X
for xfs:           xfs_check /dev/...

(xfs has a separate repair utility).
If all is well you can try:
mount -o remount,rw /dev/... /Shares

(Assuming the mount FS is /Shares, and not a symbolic link to the real fs.)
0
 
LVL 10

Expert Comment

by:pfrancois
ID: 36715620
I rather think the Windows server is not reachable over the LAN, or its IP address has changed or its drive is no longer shared correctly.
0
 
LVL 10

Expert Comment

by:pfrancois
ID: 36715627
What are the IP addresses of the Windows 2003 and the Ubuntu systems?
0
 

Author Comment

by:jparker81
ID: 36717477
The IP address for the Linux server is 10.0.0.14 the WIndows server is 10.0.0.22
The /Shares file system is on the Windows 2003 server.
0
 
LVL 39

Expert Comment

by:noci
ID: 36718181
So on 10.0.0.14/brown? you get an rsync request (from 10.0.0.22) to write to the /Shares path?

Does the user that logs on have write access to the share?
Obviously there is no write access on the /Shares path.

can you supply the output of:
mount
0
 

Author Comment

by:jparker81
ID: 36718627
Output of mount. Output of mount
0
 

Author Comment

by:jparker81
ID: 36815766
BTW the backups are be written to stor/md1 and stor/md2

Thanks!
0
 
LVL 39

Expert Comment

by:noci
ID: 36815817
Does a /Shares exist somehow on that system?

Is the rsyncd using the right directory (/stor/md1 & /stor/md2)?  It looks lik it tries /Shares maybe ./Shares was meant?
Plase verify the rsync command on the other system...?
0
 

Author Comment

by:jparker81
ID: 36815926
/Shares is on the windows server.
0
 
LVL 39

Expert Comment

by:noci
ID: 36816006
I understand, but it looks like there is a symlink (or other directory /Shares) on the Ubuntu, the the error is that Content of /Shares can not be written...

Sep 26 14:10:52 brown rsyncd[932]: rsync: readlink_stat("/Shares/Temp" (in redMon)) failed: Input/output error (5)
Sep 26 14:10:52 brown rsyncd[932]: rsync: failed to set times on "/Shares/." (in redMon): Read-only file system (30)
Sep 26 14:10:52 brown rsyncd[932]: rsync: failed to set times on "/Shares/ACH" (in redMon): Read-only file system (30)
Sep 26 14:10:52 brown rsyncd[932]: rsync: recv_generator: failed to stat "/Shares/ACH/ALAMON0074" (in redMon): Input/output error (5)

So there is some directory, which is unwriteable (ownership...)
if the rsynd runs in a chroot environment then /Shares is relative to that root).

the only filesystem that can becom read-only is your / filesystem after there are errors.
But the information doesn't add up.
Can you give the command line for the rsyncd  (ubuntu) and the rsync command (Windows)?
0
 

Author Comment

by:jparker81
ID: 36816168
Okay on the windows server rsync is running via DeltaCopy. The target server is brown. Don't know it this helps but it seems to be able to connect to the brown server.
0
 

Author Comment

by:jparker81
ID: 36816433
Here is the command used to invoke rsync from the windows server as well as the output.  OutputFromWindowsSideRsync.txt
0
Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

 
LVL 10

Expert Comment

by:pfrancois
ID: 36817510
The problem is at the Windows side: the process on the Windows server executing rsync.exe has no (more) write access rights on the "d:\Shares" directory. Check the permissions on the Windows side.
0
 
LVL 39

Assisted Solution

by:noci
noci earned 500 total points
ID: 36818406
I disagree, the problem is on the ubuntu side...

 rsync.exe  -v -rlt -z --delete "/cygdrive/D/Shares/" "brown.alamon.com::redThu/Shares/"

Ok, so this is rsync
-v   - verbose (more messages to stdout..)
-r   - recorsive
-l   - copy links as symlinks
-t   - preserve times (modify, create etc.)
-z   - compress all data in transit
--delete   - if files exist remotely that don't exist localy remove them
from: /cygdrive/D/Shares   (d:\Shares\..."
to: system brown.alamon.com  to directory :redThu/Shares/...
The username is the same on both systems, and a ssh certificate is used for authentication

the :: syntax means that there is a "module" (see section in rsync.conf on ubuntu)
The module is name   redThu.    So on the ubuntu system, check /etc/rsyncd.conf
the section [redThu] contains the path where the data REALLY goes to.. should be /stor/md1 /stor/md2...

The error messages say:
rsync: readlink_stat("/Shares/Temp" (in redThu)) failed: Input/output error (5)
IO error encountered -- skipping file deletion
./
rsync: failed to set times on "/Shares/." (in redThu): Read-only file system (30)
ACH/
rsync: failed to set times on "/Shares/ACH" (in redThu): Read-only file system (30)

So (in the module) redThu write actions fail...

Error 5 in Unix = generic IO error
Error 30 in Unix is Read-only filesystem...

So the question is what is in /etc/rsyncd.conf
please not that you may need to remove "authorized user" entries to prevent username leaking.
0
 
LVL 10

Expert Comment

by:pfrancois
ID: 36890183
@noci: you are right. Sorry. I was confused by the fact I didn't see any "/Share" directory in the output of the mounted systems of the Ubuntu side. After reading better the logs, I see the problem is at the other side.
0
 
LVL 10

Expert Comment

by:pfrancois
ID: 36890215
@noci: Since you speak about certificates, I suppose rsync in this case uses ssh for connecting to the remote machine.

I see that the Windows machine has Cygwin installed. I would like to know the output of the command, issued from the bash command line on Windows:

ssh brown@10.0.0.14 cat /etc/rsyncd.conf

Open in new window

0
 
LVL 39

Expert Comment

by:noci
ID: 36890459
Certificates might not be necessary when using modules. It depends on the module definition...
I rather forgot to remove that detail before stating my question.
0
 
LVL 39

Expert Comment

by:noci
ID: 36891009
Modules also do chroot to the file system that is mentioned in the profile of that module.
So all references are local to that module.
0
 

Author Comment

by:jparker81
ID: 36892588
0
 
LVL 39

Expert Comment

by:noci
ID: 36892734
No user spec?,==> so no authentication needed.

Did only that rsync fail, or are the other weekdays failing too?
does the rsync to redThu still fail?

what does ls -la /stor/md1/redThu
tell?
0
 

Author Comment

by:jparker81
ID: 36892777
0
 
LVL 39

Expert Comment

by:noci
ID: 36894895
Note there are several questions below..

AFAICT, the /stor/md1 was readonly at the moment this went wrong.
The error message are reliable in that respect.

I can reproduce the exact messages when mounting a readonly disk and use that as target
(esp when this is not the first update).

Timestamp updates fail with IO error, file write fail with Filesystem Readonly.
The stor/md1 stor/md2 do they get exchanged / remounted ?
Did only that rsync fail, or are the other weekdays failing too?
does the rsync to redThu still fail?
0
 

Author Comment

by:jparker81
ID: 36903659
I rebooted the machine and recieved an error: "Ureadahead error main process terminated stat 4 (1194 & 1208)

Then I pressed alt + F2 continued with boot. Attempted to run rysnc again and voila it worked? Do you think this was related to the Ureadahead error?
0
 
LVL 39

Accepted Solution

by:
noci earned 500 total points
ID: 36903987
No ureadahead is a process that tries to read all scripting & programs that are used later in the boot process to speed up the bootprocess.
By reading the scripts they get loaded in the system & filesystem caches, reducing the delay at a later stage.
So no it shouldn't be related. It can also be used to setup that process for next reboots.

This is the product description.   ( https://launchpad.net/ubuntu/+source/ureadahead )
Unless it causes a startup procedure to be skipped I cant see that is could hurt.

0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

The purpose of this article is to demonstrate how we can upgrade Python from version 2.7.6 to Python 2.7.10 on the Linux Mint operating system. I am using an Oracle Virtual Box where I have installed Linux Mint operating system version 17.2. Once yo…
How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
To efficiently enable the rotation of USB drives for backups, storage pools need to be created. This way no matter which USB drive is installed, the backups will successfully write without any administrative intervention. Multiple USB devices need t…
This tutorial will walk an individual through the steps necessary to install and configure the Windows Server Backup Utility. Directly connect an external storage device such as a USB drive, or CD\DVD burner: If the device is a USB drive, ensure i…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

21 Experts available now in Live!

Get 1:1 Help Now