Unable To Create Archive Log File

Almost three hundred gigabytes available at the destinations /archlog1 and /archlog2:

netapp03:/bbproddb1_archlog1
                      300G   32G  269G  11% /archlog1
netapp04:/bbproddb1_rman
                      1.2T  549G  578G  49% /rman
netapp03:/bbproddb1_archlog2
                      300G   32G  269G  11% /archlog2
but still:

Unable to create archive log file '/archlog2/BB60/1_317237_786457154.arc'
ARC3: Error 19504 Creating archive log file to '/archlog2/BB60/1_317237_786457154.arc'
Sat Sep 19 05:08:20 2015
Unable to create archive log file '/archlog1/BB60/1_317234_786457154.arc'
Sat Sep 19 05:08:20 2015
Unable to create archive log file '/archlog1/BB60/1_317236_786457154.arc'
ARCb: Error 19504 Creating archive log file to '/archlog1/BB60/1_317234_786457154.arc'
ARC1: Error 19504 Creating archive log file to '/archlog1/BB60/1_317236_786457154.arc'
Sat Sep 19 05:08:20 2015
Unable to create archive log file '/archlog1/BB60/1_317235_786457154.arc'
ARC0: Error 19504 Creating archive log file to '/archlog1/BB60/1_317235_786457154.arc'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance BB60 - Archival Error
ORA-16038: log 3 sequence# 317234 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 3 thread 1: '/redo1/BB60/redo03a.log'
ORA-00312: online log 3 thread 1: '/redo2/BB60/redo03b.log'
Sat Sep 19 05:08:20 2015
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance BB60 - Archival Error


Any ideas why?
xoxomosAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

xoxomosAuthor Commented:
Running 11.2.0.4 in RH 5.7
johnsoneSenior Oracle DBACommented:
Those messages appear to be in the alert log.  There should be at least one trace file for the archiver which should have additional information.

It also appears that you are writing to networked storage, did you loose connectivity?
slightwv (䄆 Netminder) Commented:
remove archived redo logs that are no longer needed
or
increase the value of the db_recovery_file_dest_size spfile paarameter.
Determine the Perfect Price for Your IT Services

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden with our free interactive tool and use it to determine the right price for your IT services. Download your free eBook now!

xoxomosAuthor Commented:
Thanks.  There was several hundred gigabytes available at log_archive_dest1 and log_archive_dest2.  How would removing  archived redo logs change the situation?

Ah yes, look for trace file.  Yes right now it does appear network is somehow at least part of the problem.
slightwv (䄆 Netminder) Commented:
>>How would removing  archived redo logs change the situation?

You need to "properly" remove them not just delete them from the file system.

If would help because if would tell Oracle the available space is less than  db_recovery_file_dest_size.

Space on the file system doesn't matter.  It is how much space Oracle thinks is available that it can use and is it below  db_recovery_file_dest_size.

This keeps Oracle from being able to take over and fill up a file system.  Think of it as a "quota".
xoxomosAuthor Commented:
By "properly" do you mean a 'DELETE archivelog all completed before .....?
Is there a way to tell how much Oracle 'thinks' is available ?
slightwv (䄆 Netminder) Commented:
>>By "properly" do you mean a 'DELETE archivelog all completed before .....?

That looks like an RMAN command.  If so, yes, "properly" means using RMAN if you are using RMAN as a backup.

If you are using something else for a backup, then you need to use tools for it.

As for that specific command you posted, maybe?  There are MANY ways to delete archived redo logs using RMAN.  I cannot tell you the correct command to use because I have no way of knowing your disaster recovery requirements and/or backup strategy.

The command you posted might delete some files that you shouldn't.  The DBA should know the backup strategy and what should and should not be able to be deleted.  Check with them.

>>Is there a way to tell how much Oracle 'thinks' is available ?

Maybe but I don't know how.  I've never needed to figure it out.  From the error message you posted, Oracle knows there isn't enough to create one more archived redo log.


Is there any reason you just don't increase db_recovery_file_dest_size?  You claim there is plenty of disk space available.  Just tell Oracle it can use more and everything will take off and start running again.
slightwv (䄆 Netminder) Commented:
I should also STRESS the importance of archived redo logs as part of a backup strategy.  If you are missing just ONE of them that is needed for recovery, you CANNOT completely recover the database.  You cannot skip over one.

Therefore I consider them the most important files to have.  DO NOT just delete them to free up space.  You need to understand when you can delete them to your comfort level.  I keep several copies of each on several different tapes.

Before you ask:  No, I cannot tell you what you need to do.  Every DBA and database has a different disaster recover plan/method they follow.  What works for one, will likely be different from another.
xoxomosAuthor Commented:
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /rman
db_recovery_file_dest_size           big integer 900G
There's over 500G free space on /rman.  The backup has a delete command that I believe takes care of archived logs
(database include current controlfile);
delete noprompt obsolete;
That is why i asked if there was a way to tell how much Oracle thinks it has available.  Since you mentioned it i'm very inclined to believe one possibility MAY be something off regarding what is available and what Oracle believes is available.
slightwv (䄆 Netminder) Commented:
It appears you can use  v$recovery_file_dest to take a look at what is used and available.

http://docs.oracle.com/cd/E11882_01/server.112/e24448/dynviews_2125.htm#REFRN30304

There also appears to be several pre-created scripts out there if you look for them.

There are probably some RMAN commands that will also provide the information.

>>There's over 500G free space on /rman.

Again, it isn't about the amount of free space on disk.  Stop trying to make it say something it isn't saying.

If db_recovery_file_dest_size is set to 900G then Oracle thinks it has used 900G.  You cannot archive another redo log until you tell Oracle there is free space there.  Again NOT free disk space!!!!!!!!!!!!

To do this:
allocate more space to db_recovery_file_dest_size
or
using RMAN commands, delete some archived redo logs

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
xoxomosAuthor Commented:
Thanks.  I'll take a look at that view you mentioned should this condition occur in the future.  
The Oracle support conclusion, which i would not have dared given management on my own.  :-)
"
So it seems some problem in underlying IO sub-system was preventing archive destinations to be available for archiving which cause the errors.

Note: We can see you are using VMWare platform. Oracle does not yet certify it for any oracle product since there are still server resources utilization problem between them. "

Worded much better than my just saying there is plenty room to write on so either NetApp or VMWare must be having problems finding it. :-)
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Oracle Database

From novice to tech pro — start learning today.