Solved

Error reported: No more data is on the tape

Posted on 2012-03-22
15
997 Views
Last Modified: 2012-03-29
We're trying to do an NDMP backup of nearly 15TB and it's failing.
Backup Server: Backup Exec 2010 R3 SP2 w/ latest hot fixes
Source: NetApp FAS3240 running 8.0.2 7-Mode
Destination:  IBM Ultrium-HH5 running BBN3 firmware

At some point after a few days, the backup just suddenly stops and Backup Exec will list the now as "Not appendable (End marker unreadable)".

I've tried rebooting the tape library and BE server, making sure there's not outstanding NDMP sessions using, and making sure I have enough tapes to accommodate the backup.

Things I haven't tried is:
- set "ndmpd.tcpnodelay.enable" to ON

But besides that, are there anything else I should look at?

NDMP Log:
NDMP Log Message: DUMP: creating "/vol/data/../snapshot_for_backup.2745" snapshot.
NDMP Log Message: DUMP: Using Full Volume Dump
NDMP Log Message: DUMP: Date of this level 1 dump: Tue Mar 20 14:46:58 2012.
NDMP Log Message: DUMP: Date of last level 0 dump: the epoch.
NDMP Log Message: DUMP: Dumping /vol/data to NDMP connection
NDMP Log Message: DUMP: mapping (Pass I)[regular files]
NDMP Log Message: DUMP: mapping (Pass II)[directories]
NDMP Log Message: DUMP: estimated 11533199972 KB.
NDMP Log Message: DUMP: dumping (Pass III) [directories]
NDMP Log Message: DUMP: Tue Mar 20 15:33:06 2012 : We have written 20679 KB.
NDMP Log Message: DUMP: Tue Mar 20 15:38:06 2012 : We have written 377185 KB.
NDMP Log Message: DUMP: Tue Mar 20 15:43:06 2012 : We have written 689998 KB.
NDMP Log Message: DUMP: Tue Mar 20 15:48:06 2012 : We have written 994199 KB.
NDMP Log Message: DUMP: Tue Mar 20 15:53:06 2012 : We have written 1349033 KB.
NDMP Log Message: DUMP: Tue Mar 20 15:58:06 2012 : We have written 1565533 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:03:06 2012 : We have written 1787571 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:08:08 2012 : We have written 1846144 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:13:08 2012 : We have written 1891033 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:18:08 2012 : We have written 1938131 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:23:08 2012 : We have written 1973194 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:28:08 2012 : We have written 2005858 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:33:08 2012 : We have written 2034443 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:38:08 2012 : We have written 2063978 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:43:08 2012 : We have written 2089131 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:48:08 2012 : We have written 2113319 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:53:08 2012 : We have written 2135828 KB.
NDMP Log Message: DUMP: Tue Mar 20 16:58:08 2012 : We have written 2157031 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:03:08 2012 : We have written 2176718 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:08:08 2012 : We have written 2198482 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:13:08 2012 : We have written 2220021 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:18:09 2012 : We have written 2240852 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:23:09 2012 : We have written 2263138 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:28:09 2012 : We have written 2285604 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:33:09 2012 : We have written 2308516 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:38:09 2012 : We have written 2331273 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:43:09 2012 : We have written 2353593 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:48:09 2012 : We have written 2374218 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:53:09 2012 : We have written 2396929 KB.
NDMP Log Message: DUMP: Tue Mar 20 17:59:38 2012 : We have written 2415092 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:04:38 2012 : We have written 2463494 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:09:38 2012 : We have written 2523142 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:14:38 2012 : We have written 2621410 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:19:38 2012 : We have written 2729913 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:24:38 2012 : We have written 2821808 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:29:38 2012 : We have written 2914044 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:34:38 2012 : We have written 2991323 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:39:38 2012 : We have written 3048181 KB.
NDMP Log Message: DUMP: dumping (Pass IV) [regular files]
NDMP Log Message: DUMP: Tue Mar 20 18:44:38 2012 : We have written 6723662 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:49:38 2012 : We have written 13386217 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:54:38 2012 : We have written 19543302 KB.
NDMP Log Message: DUMP: Tue Mar 20 18:59:38 2012 : We have written 26125548 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:04:38 2012 : We have written 33515313 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:09:38 2012 : We have written 40799913 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:14:38 2012 : We have written 47998329 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:19:38 2012 : We have written 54176878 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:24:38 2012 : We have written 59313100 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:29:38 2012 : We have written 64078188 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:34:38 2012 : We have written 71023948 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:39:38 2012 : We have written 77431341 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:44:38 2012 : We have written 82348572 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:49:38 2012 : We have written 87937272 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:54:38 2012 : We have written 95187183 KB.
NDMP Log Message: DUMP: Tue Mar 20 19:59:38 2012 : We have written 102359135 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:04:38 2012 : We have written 107927385 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:09:38 2012 : We have written 113215693 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:14:38 2012 : We have written 118522173 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:19:38 2012 : We have written 123622733 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:24:38 2012 : We have written 128822770 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:29:38 2012 : We have written 134765939 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:34:38 2012 : We have written 143566907 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:39:38 2012 : We have written 151478146 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:44:38 2012 : We have written 157447956 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:49:38 2012 : We have written 165152529 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:54:38 2012 : We have written 172932696 KB.
NDMP Log Message: DUMP: Tue Mar 20 20:59:38 2012 : We have written 179020818 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:04:38 2012 : We have written 185116068 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:09:38 2012 : We have written 190790123 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:14:38 2012 : We have written 196962129 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:19:38 2012 : We have written 202991520 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:24:38 2012 : We have written 208538913 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:29:38 2012 : We have written 215305814 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:34:38 2012 : We have written 221195143 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:39:38 2012 : We have written 227753842 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:44:38 2012 : We have written 234135037 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:49:38 2012 : We have written 240708021 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:54:38 2012 : We have written 246849100 KB.
NDMP Log Message: DUMP: Tue Mar 20 21:59:38 2012 : We have written 252991630 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:04:38 2012 : We have written 259461109 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:09:38 2012 : We have written 265366502 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:14:38 2012 : We have written 272113608 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:19:38 2012 : We have written 280238290 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:24:38 2012 : We have written 287858278 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:29:38 2012 : We have written 294177496 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:34:38 2012 : We have written 302115878 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:39:38 2012 : We have written 309589824 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:44:38 2012 : We have written 316220114 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:49:38 2012 : We have written 323890850 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:54:38 2012 : We have written 333277664 KB.
NDMP Log Message: DUMP: Tue Mar 20 22:59:38 2012 : We have written 342425678 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:04:38 2012 : We have written 351875239 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:09:38 2012 : We have written 358681692 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:14:38 2012 : We have written 366348604 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:19:38 2012 : We have written 372570066 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:24:38 2012 : We have written 378844019 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:29:38 2012 : We have written 385337029 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:34:38 2012 : We have written 391654147 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:39:38 2012 : We have written 398114324 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:44:38 2012 : We have written 405258246 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:49:38 2012 : We have written 413284049 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:54:38 2012 : We have written 419646803 KB.
NDMP Log Message: DUMP: Tue Mar 20 23:59:38 2012 : We have written 425830161 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:04:38 2012 : We have written 430422974 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:09:38 2012 : We have written 433837777 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:14:38 2012 : We have written 437522341 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:19:38 2012 : We have written 444083276 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:24:38 2012 : We have written 450162970 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:29:38 2012 : We have written 455212894 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:34:38 2012 : We have written 458524843 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:39:38 2012 : We have written 463254114 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:44:38 2012 : We have written 469297385 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:49:38 2012 : We have written 475290895 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:54:38 2012 : We have written 480428977 KB.
NDMP Log Message: DUMP: Wed Mar 21 00:59:38 2012 : We have written 484593286 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:04:38 2012 : We have written 489112645 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:09:38 2012 : We have written 494742348 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:14:38 2012 : We have written 501423718 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:19:38 2012 : We have written 508372364 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:24:38 2012 : We have written 515531126 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:29:38 2012 : We have written 522958853 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:34:38 2012 : We have written 530518931 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:39:38 2012 : We have written 535658526 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:44:38 2012 : We have written 540968764 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:49:38 2012 : We have written 547214155 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:54:38 2012 : We have written 555323599 KB.
NDMP Log Message: DUMP: Wed Mar 21 01:59:38 2012 : We have written 562786410 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:04:38 2012 : We have written 568193528 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:09:38 2012 : We have written 574067782 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:14:38 2012 : We have written 580108452 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:19:38 2012 : We have written 585900692 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:24:38 2012 : We have written 591860910 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:29:38 2012 : We have written 598105775 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:34:38 2012 : We have written 605684984 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:39:38 2012 : We have written 611896775 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:44:38 2012 : We have written 618027371 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:49:38 2012 : We have written 623815800 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:54:38 2012 : We have written 629396143 KB.
NDMP Log Message: DUMP: Wed Mar 21 02:59:38 2012 : We have written 635075142 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:04:38 2012 : We have written 641293687 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:09:38 2012 : We have written 646833250 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:14:38 2012 : We have written 652879082 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:19:38 2012 : We have written 659027151 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:24:38 2012 : We have written 666715400 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:29:38 2012 : We have written 673422571 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:34:38 2012 : We have written 679005690 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:39:38 2012 : We have written 684722108 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:44:38 2012 : We have written 690682118 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:49:38 2012 : We have written 696645350 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:54:38 2012 : We have written 702664884 KB.
NDMP Log Message: DUMP: Wed Mar 21 03:59:38 2012 : We have written 708448536 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:04:38 2012 : We have written 714506845 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:09:38 2012 : We have written 720785317 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:14:38 2012 : We have written 729837772 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:19:38 2012 : We have written 738796360 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:24:38 2012 : We have written 747963695 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:29:38 2012 : We have written 757068690 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:34:38 2012 : We have written 766108102 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:39:38 2012 : We have written 773412293 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:44:38 2012 : We have written 779530759 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:49:38 2012 : We have written 786046470 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:54:38 2012 : We have written 792184090 KB.
NDMP Log Message: DUMP: Wed Mar 21 04:59:38 2012 : We have written 798634971 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:04:38 2012 : We have written 805232051 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:09:38 2012 : We have written 809481499 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:14:38 2012 : We have written 816991308 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:19:38 2012 : We have written 823332785 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:24:38 2012 : We have written 829771691 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:29:38 2012 : We have written 836167309 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:34:38 2012 : We have written 842421844 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:39:38 2012 : We have written 848521609 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:44:38 2012 : We have written 855061669 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:49:38 2012 : We have written 861648250 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:54:38 2012 : We have written 867939799 KB.
NDMP Log Message: DUMP: Wed Mar 21 05:59:38 2012 : We have written 874575123 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:04:38 2012 : We have written 879656994 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:09:38 2012 : We have written 884260791 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:14:38 2012 : We have written 888480494 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:19:38 2012 : We have written 892947946 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:24:38 2012 : We have written 897843741 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:29:38 2012 : We have written 902824763 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:34:38 2012 : We have written 908051975 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:39:38 2012 : We have written 913558181 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:44:38 2012 : We have written 919017260 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:49:38 2012 : We have written 924759200 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:54:38 2012 : We have written 930598662 KB.
NDMP Log Message: DUMP: Wed Mar 21 06:59:38 2012 : We have written 936315208 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:04:38 2012 : We have written 942167000 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:09:38 2012 : We have written 948095463 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:14:38 2012 : We have written 954243238 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:19:38 2012 : We have written 960241907 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:24:38 2012 : We have written 966975344 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:29:38 2012 : We have written 972954852 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:34:38 2012 : We have written 978809937 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:39:38 2012 : We have written 984609285 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:44:38 2012 : We have written 990733169 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:49:38 2012 : We have written 996732319 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:54:38 2012 : We have written 1002748691 KB.
NDMP Log Message: DUMP: Wed Mar 21 07:59:38 2012 : We have written 1009327668 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:04:38 2012 : We have written 1015472125 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:09:38 2012 : We have written 1020969762 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:14:38 2012 : We have written 1027094836 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:19:38 2012 : We have written 1032991491 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:24:38 2012 : We have written 1039147143 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:29:38 2012 : We have written 1045429996 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:34:38 2012 : We have written 1052312362 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:39:38 2012 : We have written 1058791480 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:44:38 2012 : We have written 1065017911 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:49:38 2012 : We have written 1071492622 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:54:38 2012 : We have written 1077954903 KB.
NDMP Log Message: DUMP: Wed Mar 21 08:59:38 2012 : We have written 1083717573 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:14:15 2012 : We have written 1085989317 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:19:15 2012 : We have written 1094454367 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:24:59 2012 : We have written 1094800710 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:29:59 2012 : We have written 1103003143 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:36:03 2012 : We have written 1104297414 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:41:03 2012 : We have written 1110273311 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:46:03 2012 : We have written 1116718319 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:51:03 2012 : We have written 1123367110 KB.
NDMP Log Message: DUMP: Wed Mar 21 09:56:03 2012 : We have written 1129738432 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:01:03 2012 : We have written 1136173045 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:06:03 2012 : We have written 1139466255 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:11:03 2012 : We have written 1143197005 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:16:03 2012 : We have written 1147222770 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:21:03 2012 : We have written 1151180360 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:26:03 2012 : We have written 1155024086 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:31:03 2012 : We have written 1159148636 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:36:03 2012 : We have written 1163440120 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:41:03 2012 : We have written 1168582108 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:48:42 2012 : We have written 1170736391 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:53:42 2012 : We have written 1176301745 KB.
NDMP Log Message: DUMP: Wed Mar 21 10:58:42 2012 : We have written 1181815307 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:03:42 2012 : We have written 1187207108 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:08:42 2012 : We have written 1192693690 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:13:42 2012 : We have written 1198804966 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:18:42 2012 : We have written 1204111796 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:23:42 2012 : We have written 1210104389 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:28:42 2012 : We have written 1216563361 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:33:42 2012 : We have written 1222266854 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:38:42 2012 : We have written 1228474094 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:43:42 2012 : We have written 1234661085 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:48:42 2012 : We have written 1240172103 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:53:42 2012 : We have written 1246102365 KB.
NDMP Log Message: DUMP: Wed Mar 21 11:58:42 2012 : We have written 1251840711 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:03:42 2012 : We have written 1257261494 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:08:42 2012 : We have written 1260752559 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:13:42 2012 : We have written 1264463392 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:18:42 2012 : We have written 1268602347 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:23:42 2012 : We have written 1272291379 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:28:42 2012 : We have written 1276071499 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:33:42 2012 : We have written 1280287030 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:38:42 2012 : We have written 1284469380 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:43:42 2012 : We have written 1289602114 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:48:42 2012 : We have written 1294462246 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:53:42 2012 : We have written 1298932649 KB.
NDMP Log Message: DUMP: Wed Mar 21 12:58:42 2012 : We have written 1303383146 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:03:42 2012 : We have written 1308128820 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:08:42 2012 : We have written 1312994821 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:13:42 2012 : We have written 1317846452 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:18:42 2012 : We have written 1323172549 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:23:42 2012 : We have written 1331522127 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:28:42 2012 : We have written 1339822651 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:33:42 2012 : We have written 1348303348 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:38:42 2012 : We have written 1356708949 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:43:42 2012 : We have written 1365064413 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:48:42 2012 : We have written 1373632708 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:53:42 2012 : We have written 1382058848 KB.
NDMP Log Message: DUMP: Wed Mar 21 13:58:42 2012 : We have written 1390555396 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:03:42 2012 : We have written 1398282187 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:08:42 2012 : We have written 1404178370 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:13:42 2012 : We have written 1408139886 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:18:42 2012 : We have written 1411396656 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:23:42 2012 : We have written 1415998136 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:28:42 2012 : We have written 1420331918 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:33:42 2012 : We have written 1424492452 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:38:42 2012 : We have written 1428799142 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:43:42 2012 : We have written 1431672254 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:48:42 2012 : We have written 1436753139 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:53:42 2012 : We have written 1441531933 KB.
NDMP Log Message: DUMP: Wed Mar 21 14:58:42 2012 : We have written 1449785540 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:03:42 2012 : We have written 1456853445 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:08:42 2012 : We have written 1462971053 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:13:42 2012 : We have written 1469328279 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:18:42 2012 : We have written 1475912415 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:23:42 2012 : We have written 1482215468 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:28:42 2012 : We have written 1488689576 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:33:42 2012 : We have written 1495481372 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:38:42 2012 : We have written 1502657284 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:43:42 2012 : We have written 1510373431 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:48:42 2012 : We have written 1515987119 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:53:42 2012 : We have written 1519374903 KB.
NDMP Log Message: DUMP: Wed Mar 21 15:58:42 2012 : We have written 1524553220 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:03:42 2012 : We have written 1532432260 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:08:42 2012 : We have written 1540363950 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:13:42 2012 : We have written 1548765335 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:18:42 2012 : We have written 1557097839 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:23:42 2012 : We have written 1565378214 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:28:42 2012 : We have written 1573892864 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:33:42 2012 : We have written 1582264372 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:38:42 2012 : We have written 1590879478 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:43:42 2012 : We have written 1597882141 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:48:42 2012 : We have written 1600938466 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:53:42 2012 : We have written 1603990374 KB.
NDMP Log Message: DUMP: Wed Mar 21 16:58:42 2012 : We have written 1607921604 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:03:42 2012 : We have written 1612062912 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:08:42 2012 : We have written 1617743458 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:13:42 2012 : We have written 1622604160 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:18:42 2012 : We have written 1631494963 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:23:42 2012 : We have written 1640187069 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:28:42 2012 : We have written 1644105244 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:33:42 2012 : We have written 1646944294 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:38:42 2012 : We have written 1651201042 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:43:42 2012 : We have written 1658626706 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:48:42 2012 : We have written 1667327370 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:53:42 2012 : We have written 1675981461 KB.
NDMP Log Message: DUMP: Wed Mar 21 17:58:42 2012 : We have written 1684845171 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:03:42 2012 : We have written 1691823230 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:08:42 2012 : We have written 1697519980 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:13:42 2012 : We have written 1703173694 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:18:42 2012 : We have written 1709024917 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:23:42 2012 : We have written 1714776005 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:28:42 2012 : We have written 1720330570 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:33:42 2012 : We have written 1726055205 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:38:42 2012 : We have written 1731219541 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:43:42 2012 : We have written 1735107511 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:48:42 2012 : We have written 1738010009 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:53:42 2012 : We have written 1741626566 KB.
NDMP Log Message: DUMP: Wed Mar 21 18:58:42 2012 : We have written 1744791909 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:03:42 2012 : We have written 1747460169 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:08:42 2012 : We have written 1750793093 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:13:42 2012 : We have written 1753588338 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:18:42 2012 : We have written 1757133211 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:23:42 2012 : We have written 1759846742 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:28:42 2012 : We have written 1762510547 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:33:42 2012 : We have written 1768017621 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:38:42 2012 : We have written 1776403632 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:43:42 2012 : We have written 1784979453 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:48:42 2012 : We have written 1789240837 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:53:42 2012 : We have written 1792638522 KB.
NDMP Log Message: DUMP: Wed Mar 21 19:58:42 2012 : We have written 1795106905 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:03:42 2012 : We have written 1798101392 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:08:42 2012 : We have written 1801113156 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:16:36 2012 : We have written 1803244741 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:21:36 2012 : We have written 1805922246 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:26:36 2012 : We have written 1813156067 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:31:36 2012 : We have written 1817088474 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:36:36 2012 : We have written 1820941694 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:45:04 2012 : We have written 1821686788 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:50:04 2012 : We have written 1824945723 KB.
NDMP Log Message: DUMP: Wed Mar 21 20:56:23 2012 : We have written 1825487621 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:01:23 2012 : We have written 1826613209 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:07:00 2012 : We have written 1828221124 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:12:04 2012 : We have written 1829917700 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:17:38 2012 : We have written 1831352001 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:22:58 2012 : We have written 1831850244 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:28:20 2012 : We have written 1832150983 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:34:01 2012 : We have written 1832468870 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:39:27 2012 : We have written 1832695815 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:44:51 2012 : We have written 1832882311 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:51:09 2012 : We have written 1832983172 KB.
NDMP Log Message: DUMP: Wed Mar 21 21:56:23 2012 : We have written 1833069637 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:01:39 2012 : We have written 1833166788 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:06:56 2012 : We have written 1833280071 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:13:29 2012 : We have written 1833363653 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:18:29 2012 : We have written 1835984134 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:23:29 2012 : We have written 1836720078 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:28:29 2012 : We have written 1838662960 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:33:29 2012 : We have written 1841154307 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:38:29 2012 : We have written 1844267956 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:43:29 2012 : We have written 1846644658 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:48:29 2012 : We have written 1849497025 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:53:29 2012 : We have written 1853505221 KB.
NDMP Log Message: DUMP: Wed Mar 21 22:58:29 2012 : We have written 1857904274 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:03:29 2012 : We have written 1861611849 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:08:29 2012 : We have written 1865377755 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:13:29 2012 : We have written 1869262736 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:19:23 2012 : We have written 1875688838 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:24:46 2012 : We have written 1876136647 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:30:02 2012 : We have written 1876420356 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:35:20 2012 : We have written 1876756804 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:40:20 2012 : We have written 1878354467 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:45:33 2012 : We have written 1880420614 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:50:49 2012 : We have written 1882732292 KB.
NDMP Log Message: DUMP: Wed Mar 21 23:56:47 2012 : We have written 1884369476 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:02:44 2012 : We have written 1884690501 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:07:44 2012 : We have written 1886224913 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:12:46 2012 : We have written 1888686086 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:18:58 2012 : We have written 1888895751 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:23:58 2012 : We have written 1891541929 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:28:58 2012 : We have written 1893357898 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:33:58 2012 : We have written 1894945806 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:38:58 2012 : We have written 1896609998 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:43:58 2012 : We have written 1898363430 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:48:58 2012 : We have written 1900860540 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:53:58 2012 : We have written 1902982737 KB.
NDMP Log Message: DUMP: Thu Mar 22 00:58:58 2012 : We have written 1904744041 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:03:58 2012 : We have written 1907026639 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:08:58 2012 : We have written 1909601001 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:13:58 2012 : We have written 1913598751 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:18:58 2012 : We have written 1920549388 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:24:45 2012 : We have written 1921465028 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:29:45 2012 : We have written 1923083335 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:36:04 2012 : We have written 1923900167 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:41:22 2012 : We have written 1924774468 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:46:49 2012 : We have written 1925584967 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:54:18 2012 : We have written 1925857158 KB.
NDMP Log Message: DUMP: Thu Mar 22 01:59:25 2012 : We have written 1926011783 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:06:16 2012 : We have written 1926147652 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:12:22 2012 : We have written 1926210375 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:18:45 2012 : We have written 1926372102 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:24:01 2012 : We have written 1926535300 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:29:11 2012 : We have written 1926714886 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:34:12 2012 : We have written 1926900612 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:39:42 2012 : We have written 1927124164 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:44:55 2012 : We have written 1927561861 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:50:28 2012 : We have written 1927898950 KB.
NDMP Log Message: DUMP: Thu Mar 22 02:55:31 2012 : We have written 1928422663 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:00:34 2012 : We have written 1928833412 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:05:36 2012 : We have written 1929909189 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:10:36 2012 : We have written 1933316518 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:15:36 2012 : We have written 1936304290 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:20:36 2012 : We have written 1939083595 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:25:36 2012 : We have written 1941614402 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:30:36 2012 : We have written 1946229470 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:35:36 2012 : We have written 1951396019 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:40:36 2012 : We have written 1954850334 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:45:36 2012 : We have written 1957696006 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:50:36 2012 : We have written 1961039566 KB.
NDMP Log Message: DUMP: Thu Mar 22 03:55:36 2012 : We have written 1964262406 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:00:36 2012 : We have written 1966598943 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:05:36 2012 : We have written 1969672948 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:10:36 2012 : We have written 1972989099 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:15:36 2012 : We have written 1981569024 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:24:36 2012 : We have written 1983928391 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:30:38 2012 : We have written 1984299975 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:36:02 2012 : We have written 1984596358 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:42:33 2012 : We have written 1984861831 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:52:01 2012 : We have written 1984951494 KB.
NDMP Log Message: DUMP: Thu Mar 22 04:57:33 2012 : We have written 1985249351 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:03:02 2012 : We have written 1985499270 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:08:27 2012 : We have written 1985741189 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:17:35 2012 : We have written 1985973510 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:23:33 2012 : We have written 1986163717 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:28:56 2012 : We have written 1986405188 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:38:26 2012 : We have written 1986624070 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:43:45 2012 : We have written 1986941509 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:48:52 2012 : We have written 1987119620 KB.
NDMP Log Message: DUMP: Thu Mar 22 05:56:18 2012 : We have written 1987306629 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:01:38 2012 : We have written 1987624132 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:07:56 2012 : We have written 1987753732 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:13:17 2012 : We have written 1988049541 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:18:46 2012 : We have written 1988364740 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:24:47 2012 : We have written 1988504388 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:30:41 2012 : We have written 1988886982 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:35:57 2012 : We have written 1989098374 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:42:09 2012 : We have written 1989348613 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:47:53 2012 : We have written 1989578181 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:53:10 2012 : We have written 1989777157 KB.
NDMP Log Message: DUMP: Thu Mar 22 06:59:01 2012 : We have written 1989980870 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:04:23 2012 : We have written 1990261252 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:09:23 2012 : We have written 1990573714 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:14:23 2012 : We have written 1990809181 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:20:28 2012 : We have written 1992844549 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:25:28 2012 : We have written 1995505028 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:30:28 2012 : We have written 1998064452 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:35:28 2012 : We have written 2001010764 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:40:28 2012 : We have written 2004153068 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:45:28 2012 : We have written 2007593485 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:50:28 2012 : We have written 2010041491 KB.
NDMP Log Message: DUMP: Thu Mar 22 07:55:28 2012 : We have written 2012401012 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:00:28 2012 : We have written 2015077994 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:05:28 2012 : We have written 2017618897 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:10:28 2012 : We have written 2020272922 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:15:28 2012 : We have written 2022583897 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:20:28 2012 : We have written 2024960967 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:25:28 2012 : We have written 2028222250 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:30:28 2012 : We have written 2030296777 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:35:28 2012 : We have written 2033416932 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:40:28 2012 : We have written 2036934744 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:45:28 2012 : We have written 2039498883 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:50:28 2012 : We have written 2041997016 KB.
NDMP Log Message: DUMP: Thu Mar 22 08:55:28 2012 : We have written 2044980442 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:00:28 2012 : We have written 2047939981 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:09:56 2012 : We have written 2048697607 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:15:31 2012 : We have written 2049084741 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:20:44 2012 : We have written 2049240069 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:31:51 2012 : We have written 2049467588 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:40:09 2012 : We have written 2049580805 KB.
NDMP Log Message: DUMP: Thu Mar 22 09:49:44 2012 : We have written 2049907206 KB.
NDMP Log Message: DUMP: Media error on tape write.
NDMP Log Message: DUMP: Thu Mar 22 09:57:15 2012 : We have written 2050248710 KB.
NDMP Log Message: DUMP: DUMP IS ABORTED
NDMP Log Message: DUMP: Deleting "/vol/data/../snapshot_for_backup.2745" snapshot.
NDMP Data Halted: Connect Error
NDMP Log Message: Connection or IO Error.
NDMP Mover Halted: Media Error
NDMP Log Message: MoveletOutput: Media Error.


================  NDMP ENVIRONMENT  ================
DMP_NAME=
DIRECT=N
VERBOSE=N
DEBUG=N
LIST=N
LIST_QTREES=N
EXTRACT=Y
RESTART=N
IGNORE_QTREES=N
MULTI_SUBTREE_NAMES=
RECOVER_FULL_PATHS=N
FILESYSTEM=/vol/data
EXTRACT_ACL=Y
DATA_BLOCK_SIZE=64
REPLICATE=N
TYPE=dump
RECURSIVE=Y
HIST=Y
LEVEL=1
EXCLUDE=
NDMP_UNICODE_FH=N
NDMP_VERSION=4
FORCE=N
DUMP_DATE=-1
UPDATE=Y
NON_QUOTA_TREE=N
NOWRITE=N
BASE_DATE=-1
NO_ACLS=N
===============  NDMP ENVIRONMENT END  ===============


netapp> options ndmpd
ndmpd.access                 all
ndmpd.authtype               challenge
ndmpd.connectlog.enabled     off
ndmpd.data_port_range        all
ndmpd.enable                 on
ndmpd.ignore_ctime.enabled   off
ndmpd.offset_map.enable      on
ndmpd.password_length        16
ndmpd.preferred_interface    disable    (value might be overwritten in takeover)
ndmpd.tcpnodelay.enable      off
0
Comment
Question by:randy915
  • 8
  • 6
15 Comments
 
LVL 47

Expert Comment

by:dlethe
ID: 37756011
There is probably no end of tape marker.  Format the tape.
0
 
LVL 1

Author Comment

by:randy915
ID: 37758055
We have a library of 48, more than half of them are having these errors; does this mean BE/tape library will still try to use these tapes which results in this error?  I don't want to do a mass quick erase unless that's the likely culprit.
0
 
LVL 47

Accepted Solution

by:
dlethe earned 500 total points
ID: 37758359
I have no idea, Randy.  Some tapes are pre-formatted, some have to be formatted by a changer, or application software.

I (my company) write low-level tape diagnostic software products for OEMs, so we see all permutations of this problem.  There are also programmatic methods to confirm such things, and get to the bottom of it all, and see how many feet down, and so on, but going down that path is well beyond scope of EE.

So I am playing the odds, format the tape and repeat.  If it solves the problem, then format the ones that failed.   Then you have to decide if it is worth the time and effort to format the ones that you haven't tried before there is a problem, or format after-the-fact, then re-run the backup.

But if you are seeing a 50% failure rate, then you are better off formatting known bad and all untested tapes.   You might google tapealert.org , which is the ANSI specification that covers errors / predictive problems with tape drives.  Maybe you will see some data there about other symptoms.  The HP Ultrium certainly is compliant with the TAPEALERT specification so you probably have software from HP that can drill into the drive.

 I'm responding with what makes sense to me as likely root cause based on my experience with tapes in general, as I have ZERO experience with HP Ultriums.
0
 
LVL 1

Author Comment

by:randy915
ID: 37758759
Thanks dlethe, very insightful.

I have removed all the bad tapes in the library and put in sufficient good tapes and will attempt the NDMP dump again, will let you  know how that goes.

Generally speaking, can I still use these bad tapes for restoration if they contain bits of incremental for other successful backup jobs (VSS-based jobs) or is the whole tape SOL?  If SOL, then there's no point in keeping them for recovery...
0
 
LVL 47

Expert Comment

by:dlethe
ID: 37759366
The tapes are NOT necessarily bad.  They probably don't have the EOT markers down which is something that just takes software to add.  Do that, and you very well may be golden.
0
 
LVL 1

Author Comment

by:randy915
ID: 37767185
I tried a 14TB backup from our other NetApp w/ TCPnodelay turned on and this one failed as well even though another much smaller volume backs up fine.

A couple tapes now have bad end markers so these failed backups seem to create them.
0
 
LVL 1

Author Comment

by:randy915
ID: 37767213
I will remove slot 10, quick erase the other 3 with end marker unreadable and try again.

----- TapeAlert Events -----

The device has reported the following TapeAlert diagnostic information.
Please contact your device manufacturer for more information regarding TapeAlerts.

Warning - The tape drive may have a hardware fault.
Check for availability of diagnostic information and run extended diagnostics if applicable.
Check the tape drive users manual for instructions on running extended diagnostic tests and retrieving diagnostic data.
Robotic Library: Spectra
Drive: LTO5
Slot: 10

Warning - A severe error occurred while reading or writing data.  This job may not have been successfully completed.
Robotic Library: Spectra
Drive: LTO5
Slot: 10

Critical - The tape is from a faulty batch or the tape drive is faulty:
1. Use a good tape to test the drive.
2. If the problem persists, call the tape drive supplier helpline.
Robotic Library: Spectra
Drive: LTO5
Slot: 10
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 
LVL 47

Expert Comment

by:dlethe
ID: 37767224
An EOT marker is created by sending a command, and can be written over.   Did you have the necessary tape formatting software to attempt to fix the problem on one of these "bad" tapes?

(Certainly it is possible you have some incompatible tapes, or ones that are such low quality that only half of them are good.  It isn't as if all tapes are of equal quality and made to spec.   If these are 3rd party tapes bought in bulk, instead of directly from NetApp, you do run risk of of getting junk.  

Make sure that you do actually have CERTIFIED make/models of tapes.  With this high-density data, there is little room for error, and as somebody who worked for a large computer distributor years ago, I know that tapes are not all the same.
0
 
LVL 1

Author Comment

by:randy915
ID: 37767638
I believe they come pre-formatted but so far I've only attempted to fix the situation with quick erase.
0
 
LVL 1

Author Comment

by:randy915
ID: 37768882
Failed again, different tape but the same error and EOT marker.
0
 
LVL 16

Expert Comment

by:Gerald Connolly
ID: 37775402
So are all these failures the same?

ie you start with a good tape (one with a EOT mark), you run the job, it runs for a while, then the job fails and then there isnt a EOT mark?

or is there an intermediate step where a job fails and then subsequent jobs fail with this error?
0
 
LVL 1

Author Comment

by:randy915
ID: 37777461
I quick erased all tapes with bad EOT so that the whole library has either good tapes with data on them and good tapes that are marked as scratch media.

I tried this about 3 times now with this scenario and they all failed.

I even disabled TCP chimney, TCP offline engine, and turned on flow control on Backup Exec's NIC as recommended by Symantec for NDMP backups.

Tape drive test using the built-in Spectra library's utilities are ok for both LTO4/5 and I've been using LTO5 exclusively for these experiments by pausing LTO4 before I kick off the backup.

This is just baffling and frustrating.
0
 
LVL 47

Expert Comment

by:dlethe
ID: 37777507
Anything tying the "bad" tapes or even the good tapes together?  There is typically a batch number or something that identifies tape runs.  Maybe just a bad batch?
0
 
LVL 47

Expert Comment

by:dlethe
ID: 37777601
Well, there is some software you can run that will dig into the tape library and report detailed error messages.  Risk is that you buy it and it turns out that the tape isn't returning anything useful.  (But the LTOs aren't exactly cheap consumer tapes, so you would think they are sending out useful info.   Plus you need to run a direct host-tape and can't get this data from within the net app).

So basically, you have to move things around and potentially buy a stand-alone controller and use different software to backup, but the answer can be obtained.  Since the message says potential hardware then I'd contact the tape vendor. maybe something you can do with LED codes or something you can download.
0
 
LVL 1

Author Closing Comment

by:randy915
ID: 37783087
All signs are pointing to a bad LTO-5 drive.  The LTO-4 is backing up the said volumes just fine via NDMP and we constantly (3-5 days) have to clean LTO-5 to keep it happy.  Awaiting vendor to come in and do further diagnosis but given the evidence it's safe to say this case is solved.

If erasing the tapes are not helping and the media + source servers are configured correctly then most likely it's the backup hardware.
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

A quick step-by-step overview of installing and configuring Carbonite Server Backup.
Create your own, high-performance VM backup appliance by installing NAKIVO Backup & Replication directly onto a Synology NAS!
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This tutorial will walk an individual through the process of configuring basic necessities in order to use the 2010 version of Data Protection Manager. These include storage, agents, and protection jobs. Launch Data Protection Manager from the deskt…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now