Link to home
Start Free TrialLog in
Avatar of hendrixl
hendrixl

asked on

All Arcserve versions walk to ROOT every 1 second when running on non-replica holding servers

This is really an informational post.  I am not sure a TID was every created to document this issue.

On non-replica holding servers an individual Arcserve server objects will walk to a server holding a copy of ROOT once every second.  This caused some serious issues in our environment when combined with a ROOT server sitting on a lower cost network segment.  Basically it creates a Denial of Service condition for the ROOT server(s).  The server(s) are unable to respond to DS requests, but are still in an UP state in DS.

eDirectory is asked to reply with a referral list, a list of addresses of who might know about the object AQ6_servername, in turn, eDirectory uses the first IP address returned to the requesting server.  Since the requesting server doesn't hold any real replicas (all referrals fail), we start by talking to servers who are in our connection table (servers that are our parents as far as DS is concerned), from the first IP address we attempt a connection and ask for a list of referrals. The referral list is passed back to the requesting server, and then back to ArcServe (the real requestor).  We do a comparison of two objects, the AQ6_servername object, and the AS_BACKUP_SERVER object.  Arcserve is only asking for referrals and not objectid's.  This request is based on some very old APIs that were never updated.

Just for an idea of the scope of the issue using a 75 non-replica holding server environment:

Arcserve is making (60 requests per minute * 60 minutes per hour *24 hours per day = 86400 requests daily ) * (75 - approx number or servers with Arcserve generating traffic) = 6,480,000 overall requests per day. Payload per request is 350 bytes per packet.
Running the calculations:
(350 bytes * 6,480,000 requests) = 2,268,000,000 total bytes or (2,268,000,000 / 1024) = 2,214,843.75 megabytes or 2,162 Gigabytes of traffic generated by Arcserve on the network daily.”

Computer Associates has been working on a patch to change the timing of the request from 1 second to a configurable parameter.  This will also change the amount of time it could take to update changes to job queues, etc.  

I don't believe CA has any immediate plans to "fix" the API calls so that a R/W replica holder in the same paritition could service this request.  We have only recently received the patch and finished testing it.  I have not been involved in the latest discussions with CA, but I believe they maintain that best practice would be to put a replica on every server running Arcserve....  not really an option for a 500 server environment like ours.

I am not certain, but believe Computer Associates is including this configurable patch in the latest Enterpise 9 update release.

lh



ASKER CERTIFIED SOLUTION
Avatar of ShineOn
ShineOn
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of hendrixl
hendrixl

ASKER

We are currently running AS7 Enterprise with the latest patches (soon to include the patch for this problem).  When they started working this issue it was still a supported product :-).  Has taken MONTHs to get a patch that works.  This issue is present in AS9 also, but I am not sure if there a patch level for AS9 that includes this fix, or if you have to request it.  Maybe they fixed it in 11, but when the problem was reported to them it was present in all shipping versions of AS that we tested.  That was in June/July of this year.

lh
BTW - the objects for AS7 Enterprise are still named AQ6 in the tree.  I see your confusion now.  Sorry I wasn't more clear.

lh
NW6 SP5.
SOLUTION
Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Here is section of a DS Trace performed in a test tree


40B7AF06:3531264:d5b505a0:030 Calling DSAResolveName conn:21 for client .[Public].
40B7AF06:3531264:d5b505a0:028 Request - (108)
0000  02 00 00 00 02 40 00 00 00 00 00 00 00 00 00 00 .....@..........
0010  00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 ................
0020  04 00 00 00 00 00 00 00 C4 00 00 00 04 00 00 00 ................
0030  07 00 00 00 6E A4 32 35 01 00 00 00 6E A4 32 35 ....n.25....n.25
0040  01 00 04 00 6E A4 32 35 01 00 24 00 6E A4 32 35 ....n.25..$.n.25
0050  01 00 44 00 6E A4 32 35 01 00 64 00 6E A4 32 35 ..D.n.25..d.n.25
0060  01 00 84 00 5D 61 B7 40 01 00 0C 00 00 00 00 00 ....]a.@........
0070  7C 00 00 00 5C 00 54 00 3D 00 54 00 45 00 53 00 |...\.T.=.T.E.S.
0080  54 00 2D 00 54 00 52 00 45 00 45 00 5C 00 4F 00 T.-.T.R.E.E.\.O.
0090  3D 00 4F 00 4E 00 45 00 5C 00 4F 00 55 00 3D 00 =.O.N.E.\.O.U.=.
00A0  45 00 53 00 54 00 5C 00 4F 00 55 00 3D 00 4F 00 E.S.T.\.O.U.=.O.
00B0  48 00 5C 00 4F 00 55 00 3D 00 53 00 56 00 43 00 H.\.O.U.=.S.V.C.
00C0  5C 00 4F 00 55 00 3D 00 53 00 56 00 52 00 5C 00 \.O.U.=.S.V.R.\.
00D0  43 00 4E 00 3D 00 41 00 51 00 36 00 5F 00 54 00 C.N.=.A.Q.6._.T.
00E0  45 00 53 00 54 00 2D 00 4E 00 57 00 36 00 00 00 E.S.T.-.N.W.6...
00F0  02 00 00 00 08 00 00 00 09 00 00 00 02 00 00 00 ................

40B7AF06:3531264:d5b505a0:028
0000  08 00 00 00 09 00 00 00                         ........

40B7AF06:3531265:d5b505a0:028 Reply - (32)
0000  06 00 00 00 40 00 00 00 78 80 00 00 01 00 00 00 ....@...x.......
0010  02 00 00 00 09 00 00 00 06 00 00 00 02 0C 36 60 ..............6`
0020  02 84 00 00 08 00 00 00 06 00 00 00 02 0C 36 60 ..............6`
0030  02 84                                           ..

40B7AF06:3531265:d0925180:030 Calling DSACompare conn:21 for client .[Public].
40B7AF06:3531265:d0925180:028 Request - (a8)
0000  00 00 00 00 78 80 00 00 01 00 00 00 0A 00 00 00 ....x...........
0010  55 00 73 00 65 00 72 00 00 00 00 00 01 00 00 00 U.s.e.r.........
0020  84 00 00 00 5C 00 54 00 3D 00 54 00 45 00 53 00 ....\.T.=.T.E.S.
0030  54 00 2D 00 54 00 52 00 45 00 45 00 5C 00 4F 00 T.-.T.R.E.E.\.O.
0040  3D 00 4F 00 4E 00 45 00 5C 00 4F 00 55 00 3D 00 =.O.N.E.\.O.U.=.
0050  45 00 53 00 54 00 5C 00 4F 00 55 00 3D 00 4F 00 E.S.T.\.O.U.=.O.
0060  48 00 5C 00 4F 00 55 00 3D 00 53 00 56 00 43 00 H.\.O.U.=.S.V.C.
0070  5C 00 4F 00 55 00 3D 00 53 00 56 00 52 00 5C 00 \.O.U.=.S.V.R.\.
0080  43 00 4E 00 3D 00 41 00 53 00 5F 00 42 00 41 00 C.N.=.A.S._.B.A.
0090  43 00 4B 00 55 00 50 00 5F 00 53 00 45 00 52 00 C.K.U.P._.S.E.R.
00A0  56 00 45 00 52 00 00 00                         V.E.R...

40B7AF06:3531265:d0925180:028 Reply - (1)
0000  01                                              .

40B7AF07:3531758:d0925180:030 Calling DSAResolveName conn:21 for client .[Public].
40B7AF07:3531758:d0925180:028 Request - (108)
0000  02 00 00 00 02 40 00 00 00 00 00 00 00 00 00 00 .....@..........
0010  00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 ................
0020  04 00 00 00 00 00 00 00 C4 00 00 00 04 00 00 00 ................
0030  07 00 00 00 6E A4 32 35 01 00 00 00 6E A4 32 35 ....n.25....n.25
0040  01 00 04 00 6E A4 32 35 01 00 24 00 6E A4 32 35 ....n.25..$.n.25
0050  01 00 44 00 6E A4 32 35 01 00 64 00 6E A4 32 35 ..D.n.25..d.n.25
0060  01 00 84 00 5D 61 B7 40 01 00 0C 00 00 00 00 00 ....]a.@........
0070  7C 00 00 00 5C 00 54 00 3D 00 54 00 45 00 53 00 |...\.T.=.T.E.S.
0080  54 00 2D 00 54 00 52 00 45 00 45 00 5C 00 4F 00 T.-.T.R.E.E.\.O.
0090  3D 00 4F 00 4E 00 45 00 5C 00 4F 00 55 00 3D 00 =.O.N.E.\.O.U.=.
00A0  45 00 53 00 54 00 5C 00 4F 00 55 00 3D 00 4F 00 E.S.T.\.O.U.=.O.
00B0  48 00 5C 00 4F 00 55 00 3D 00 53 00 56 00 43 00 H.\.O.U.=.S.V.C.
00C0  5C 00 4F 00 55 00 3D 00 53 00 56 00 52 00 5C 00 \.O.U.=.S.V.R.\.
00D0  43 00 4E 00 3D 00 41 00 51 00 36 00 5F 00 54 00 C.N.=.A.Q.6._.T.
00E0  45 00 53 00 54 00 2D 00 4E 00 57 00 36 00 00 00 E.S.T.-.N.W.6...
00F0  02 00 00 00 08 00 00 00 09 00 00 00 02 00 00 00 ................

40B7AF07:3531758:d0925180:028
0000  08 00 00 00 09 00 00 00                         ........

40B7AF07:3531759:d0925180:028 Reply - (32)
0000  06 00 00 00 40 00 00 00 78 80 00 00 01 00 00 00 ....@...x.......
0010  02 00 00 00 09 00 00 00 06 00 00 00 02 0C 36 60 ..............6`
0020  02 84 00 00 08 00 00 00 06 00 00 00 02 0C 36 60 ..............6`
0030  02 84                                           ..

40B7AF07:3531759:d5b505a0:030 Calling DSACompare conn:21 for client .[Public].
40B7AF07:3531759:d5b505a0:028 Request - (a8)
0000  00 00 00 00 78 80 00 00 01 00 00 00 0A 00 00 00 ....x...........
0010  55 00 73 00 65 00 72 00 00 00 00 00 01 00 00 00 U.s.e.r.........
0020  84 00 00 00 5C 00 54 00 3D 00 54 00 45 00 53 00 ....\.T.=.T.E.S.
0030  54 00 2D 00 54 00 52 00 45 00 45 00 5C 00 4F 00 T.-.T.R.E.E.\.O.
0040  3D 00 4F 00 4E 00 45 00 5C 00 4F 00 55 00 3D 00 =.O.N.E.\.O.U.=.
0050  45 00 53 00 54 00 5C 00 4F 00 55 00 3D 00 4F 00 E.S.T.\.O.U.=.O.
0060  48 00 5C 00 4F 00 55 00 3D 00 53 00 56 00 43 00 H.\.O.U.=.S.V.C.
0070  5C 00 4F 00 55 00 3D 00 53 00 56 00 52 00 5C 00 \.O.U.=.S.V.R.\.
0080  43 00 4E 00 3D 00 41 00 53 00 5F 00 42 00 41 00 C.N.=.A.S._.B.A.
0090  43 00 4B 00 55 00 50 00 5F 00 53 00 45 00 52 00 C.K.U.P._.S.E.R.
00A0  56 00 45 00 52 00 00 00                         V.E.R...

40B7AF07:3531759:d5b505a0:028 Reply - (1)
0000  01                                              .

40B7AF07:3531760:d5b505a0:030 Calling DSAResolveName conn:21 for client .[Public].
40B7AF07:3531760:d5b505a0:028 Request - (108)
0000  02 00 00 00 02 40 00 00 00 00 00 00 00 00 00 00 .....@..........
0010  00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 ................
0020  04 00 00 00 00 00 00 00 C4 00 00 00 04 00 00 00 ................
0030  07 00 00 00 6E A4 32 35 01 00 00 00 6E A4 32 35 ....n.25....n.25
0040  01 00 04 00 6E A4 32 35 01 00 24 00 6E A4 32 35 ....n.25..$.n.25
0050  01 00 44 00 6E A4 32 35 01 00 64 00 6E A4 32 35 ..D.n.25..d.n.25
0060  01 00 84 00 5D 61 B7 40 01 00 0C 00 00 00 00 00 ....]a.@........
0070  7C 00 00 00 5C 00 54 00 3D 00 54 00 45 00 53 00 |...\.T.=.T.E.S.
0080  54 00 2D 00 54 00 52 00 45 00 45 00 5C 00 4F 00 T.-.T.R.E.E.\.O.
0090  3D 00 4F 00 4E 00 45 00 5C 00 4F 00 55 00 3D 00 =.O.N.E.\.O.U.=.
00A0  45 00 53 00 54 00 5C 00 4F 00 55 00 3D 00 4F 00 E.S.T.\.O.U.=.O.
00B0  48 00 5C 00 4F 00 55 00 3D 00 53 00 56 00 43 00 H.\.O.U.=.S.V.C.
00C0  5C 00 4F 00 55 00 3D 00 53 00 56 00 52 00 5C 00 \.O.U.=.S.V.R.\.
00D0  43 00 4E 00 3D 00 41 00 51 00 36 00 5F 00 54 00 C.N.=.A.Q.6._.T.
00E0  45 00 53 00 54 00 2D 00 4E 00 57 00 36 00 00 00 E.S.T.-.N.W.6...
00F0  02 00 00 00 08 00 00 00 09 00 00 00 02 00 00 00 ................

40B7AF07:3531760:d5b505a0:028
0000  08 00 00 00 09 00 00 00                         ........

40B7AF07:3531760:d5b505a0:028 Reply - (32)
0000  06 00 00 00 40 00 00 00 78 80 00 00 01 00 00 00 ....@...x.......
0010  02 00 00 00 09 00 00 00 06 00 00 00 02 0C 36 60 ..............6`
0020  02 84 00 00 08 00 00 00 06 00 00 00 02 0C 36 60 ..............6`
0030  02 84                                           ..

40B7AF07:3531761:d5b505a0:030 Calling DSACompare conn:21 for client .[Public].
40B7AF07:3531761:d5b505a0:028 Request - (a8)
0000  00 00 00 00 78 80 00 00 01 00 00 00 0A 00 00 00 ....x...........
0010  55 00 73 00 65 00 72 00 00 00 00 00 01 00 00 00 U.s.e.r.........
0020  84 00 00 00 5C 00 54 00 3D 00 54 00 45 00 53 00 ....\.T.=.T.E.S.
0030  54 00 2D 00 54 00 52 00 45 00 45 00 5C 00 4F 00 T.-.T.R.E.E.\.O.
0040  3D 00 4F 00 4E 00 45 00 5C 00 4F 00 55 00 3D 00 =.O.N.E.\.O.U.=.
0050  45 00 53 00 54 00 5C 00 4F 00 55 00 3D 00 4F 00 E.S.T.\.O.U.=.O.
0060  48 00 5C 00 4F 00 55 00 3D 00 53 00 56 00 43 00 H.\.O.U.=.S.V.C.
0070  5C 00 4F 00 55 00 3D 00 53 00 56 00 52 00 5C 00 \.O.U.=.S.V.R.\.
0080  43 00 4E 00 3D 00 41 00 53 00 5F 00 42 00 41 00 C.N.=.A.S._.B.A.
0090  43 00 4B 00 55 00 50 00 5F 00 53 00 45 00 52 00 C.K.U.P._.S.E.R.
00A0  56 00 45 00 52 00 00 00                         V.E.R...

40B7AF07:3531761:d5b505a0:028 Reply - (1)
0000  01                                          
and here is a single packet with details

No. Source Destination Protocol Info
2454 CMHNV604.svr.bankone.net xx.xx.xx.xx NDS C NDS Resolve Name -> CN=AQ6_CMHNV604.
Frame 2454 (222 bytes on wire, 222 bytes captured)
Arrival Time: May 28, 2004 10:06:05.769067000
Time delta from previous packet: 71.070394000 seconds
Time since reference or first frame: 71.070394000 seconds
Frame Number: 2454
Packet Length: 222 bytes
Capture Length: 222 bytes
Ethernet II, Src: 00:02:a5:3f:4f:a4, Dst: 00:00:0c:07:ac:00
Destination: 00:00:0c:07:ac:00 xx.xx.xx.xx
Source: 00:02:a5:3f:4f:a4 (CMHNV604.svr.bankone.net)
Type: IP (0x0800)
Internet Protocol, Src Addr: CMHNV604.svr.bankone.net xx.xx.xx.xx, Dst Addr: xx.xx.xx.xx
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 208
Identification: 0xaf2e (44846)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 128
Protocol: TCP (0x06)
Header checksum: 0x85a4 (correct)
Source: CMHNV604.svr.bankone.net
Destination: xx.xx.xx.xx
Transmission Control Protocol, Src Port: 25034 (25034), Dst Port: 524 (524), Seq: 2020, Ack: 2354, Len
Source port: 25034 (25034)
Destination port: 524 (524)
Sequence number: 2020 (relative sequence number)
Next sequence number: 2188 (relative sequence number)
Acknowledgement number: 2354 (relative ack number)
Header length: 20 bytes
Flags: 0x0018 (PSH, ACK)
0... .... = Congestion Window Reduced (CWR): Not set
.0.. .... = ECN-Echo: Not set
..0. .... = Urgent: Not set
...1 .... = Acknowledgment: Set
.... 1... = Push: Set
.... .0.. = Reset: Not set
.... ..0. = Syn: Not set
.... ...0 = Fin: Not set
Window size: 3791
Checksum: 0x6806 (correct)
SEQ/ACK analysis
This is an ACK to the segment in frame: 2453
The RTT to ACK the segment was: 0.008996000 seconds
NetWare Core Protocol
NCP over IP signature: Transport is NCP (Reply) (0x744e6350)
NCP over IP length: 168
Type: Service request (0x2222)
Sequence Number: 14
Connection Number: 237
Task Number: 1
NDS Reply Buffer Size: 4096
Function: 104 (0x68), Send NDS Fragmented Request/Reply
SubFunction: 2 (0x02)
NDS Fragment Handle: 0xffffffff
NDS Fragment Size: 514
Message Size: 140
Flags: 0x00000000
NDS Verb: 1, (0x01), Resolve Name
Version: 0
Flags: 0x2062 - (Readable, Walk Tree, Dereference Alias, Prefer Referrals)
.... .... .... ..1. = Readable: True
.... .... ..1. .... = Walk Tree: True
.... .... .1.. .... = Dereference Alias: True
..1. .... .... .... = Prefer Referrals: True
Scope: 0
Name: CN=AQ6_CMHNV604.OU=B1CC.OU=OH.OU=EST.O=ONE
Communications Transports: 2
(TCP Protocol)
(UDP Protocol)
Tree Walker Transport Type: 2
(TCP Protocol)
(UDP Protocol)
0000 00 00 0c 07 ac 00 00 02 a5 3f 4f a4 08 00 45 00 .........?O...E.
0010 00 d0 af 2e 40 00 80 06 85 a4 0a 81 04 a4 41 40 ....@.........A@
0020 74 f0 61 ca 02 0c 32 95 24 bc 13 b2 de ad 50 18 t.a...2.$.....P.
0030 0e cf 68 06 00 00 74 4e 63 50 00 00 00 a8 22 22 ..h...tNcP....""
0040 0e ed 01 09 68 02 ff ff ff ff 02 02 00 00 8c 00 ....h...........
0050 00 00 00 00 00 00 01 00 00 00 00 10 00 00 00 00 ................
0060 00 00 62 20 00 00 00 00 00 00 56 00 00 00 43 00 ..b ......V...C.
0070 4e 00 3d 00 41 00 51 00 36 00 5f 00 43 00 4d 00 N.=.A.Q.6._.C.M.
0080 48 00 4e 00 56 00 36 00 30 00 34 00 2e 00 4f 00 H.N.V.6.0.4...O.
0090 55 00 3d 00 42 00 31 00 43 00 43 00 2e 00 4f 00 U.=.B.1.C.C...O.
00a0 55 00 3d 00 4f 00 48 00 2e 00 4f 00 55 00 3d 00 U.=.O.H...O.U.=.
00b0 45 00 53 00 54 00 2e 00 4f 00 3d 00 4f 00 4e 00 E.S.T...O.=.O.N.
00c0 45 00 00 00 95 d2 02 00 00 00 09 00 00 00 08 00 E...............
00d0 00 00 02 00 00 00 09 00 00 00 08 00 00 00 ..............

"I don't know if AS9 still uses the old AQ6 objects, but I'm pretty sure 11.1 doesn't - it just uses the native TSA's."

ShineOn - there are two ARCserve objects in AS9 and I am sure they exist in r11.1 as well.  They are the ARCserve Server object and the ARCserve Backup Queue object (not to be confused with a print queue).

The ARCserve Server object exists to define which server in the tree is running the ARCserve service.  The ARCserve Backup Queue object contains the backup jobs.

Thus, they really don't use the TSA's.  Now, the application running on the servers, it will access the TSA's when backing up information specific to that server (TSANDS for backing up eDirectory hosted or accessed from that server, TSA500/TSA600/TSAFS to access the NetWare volume file system and in the future - GroupWise, GWTSA to currently access the GroupWise system, and eventually TSAFAT to access the DOS boot partition).  Or is that TSADOS??? I can't remember that one.

Note:  Novell is moving the features of GWTSA into the TSAFS in future versions of the TSA and GroupWise.  It's also increasing it's functionality so that you can get better backups of GroupWise.  I expect GWTSA to cease as of GW6.5 and probably won't exist in GW7 when it ships next year.

Note:  I've seen GW7 screen shots - shared folders are more like collaboration folders now because the right-hand view won't just be files, or folders, or calendar or message threads.  You will be able to see it ALL in multiple "portlets" in the right-side view when you select a shared folder.  Exchange killer if you ask me.

I've got ARCserve r11.1 but I haven't installed it yet.  I am waiting for the first Service Pack from CA before I do so (like I did with version 9) - I avoid ALL problems by waiting for the first SP from ANY vendor - except Microsoft, I wait until the SECOND SP for applications (Office) and the THIRD SP for the operating system ;)

But I will test it next week in my lab to see if it still creates the AS NDS objects.