Exchange database reporting as full after failed backup

Create a D drive and move the database to D drive.
https://www.experts-exchange.com/articles/31688/Move-Mailboxdatabase-and-other-logs-to-different-drive-in-Exchange-2013-2016-2019.html

If you want to move mailboxes without down time, Create a database in D drive and move mailboxes to the new database. Once all mailboxes moved to new database including arbitration, Archive etc. Delete the old database. Moving arbitration mailboxes also explained in the following article.
https://www.experts-exchange.com/articles/31741/Move-mailboxes-in-bulk-or-one-by-one-to-another-database-in-Exchange-2010-2013-2016.html

Enable circular logging for cleaning up the logs.

Set-MailboxDatabase databasename -CircularLoggingEnabled $true

Select allOpen in new window

...and the following day started having incoming email issues.  

obvious back pressure
once free space on the volume where queues are located gets below a certain threshold, the transport service will shutdown causing mail flow issues

I ran a log cleanup script which did clean out about 10 gigs of logs...

what log cleanup script?  exchange transaction logs should never be touched manually

...but it did not create any additional database room.

purging logs will not touch database size
your 96mb is how much white space is in the database
if free space is that low, the database will dismount (if it hasn't done so already)

agreed you should enable circular logging to properly clean the logs and free disk space; should be disabled after finishing this exercise as it shouldn't be enabled long-term

if this is a virtual machine, add another disk and use the links Mas cited for moving your database/logs elsewhere
I don't think you will be able to do it at the moment until you resolve your disk space issue which would be to properly  remove transaction logs

once all of that is done, your exchange services are normal and mail is flowing again, should examine why backups are failing

Your database size is 158GB but your transaction log files have grown too hugh which is why disk space is a problem.

backups will fail cause due to hugh log files, these log files might not be in sequence, missing log files and corrupt log files dats why backups are failing and not purging them.

Try running disk shadow cmd line utility to purge logs using elevate cmd. But first make sure exchange vss writers are in stable state with no error. Probably by now your backups might have brought the exchange vss writers to inconsistence state.

From elevated cmd run command: vssadmin list writers

See if the exchange vss writer is stable with no error.

Then follow diskshadow

diskshadow
begin backup
add volume c: alias vss_c
create
end backup

In the above example I have assumed your exchange database and transaction log files are on same C drive. If your exchange database and log reside on different disk then you need to add one more additional line like example below:

add volume e: alias vss_e
add volume f: alias vss_f

when you run command create wait for it to execute completely and wait and once the backup starts run last cmd end backup which will flush all exchange transaction logs online and will not dismount or cause database dismount issues. Your database will still be mounted and healthy.

So this was doing it online. So lets say your exchange vss writers are not is stable state and you cannot afford to restart the information store service due to various reasons, then do this manual job below.

1. Gracefully dismount the database.
2. From cmd elevated cmd run command: eseutil /mh "database path with file name and ext."
3. Once the eseutil /mh  command runs verify if the state of database shows as clean shutdown.
4. Once verified database is in clean state. Move/Delete all exchange transaction log file by selecting all or in bunches or simply rename the log folder and replace with new log folder and make sure only EDB and content index CI folder exist in the current database location.
5. Then mount the database to create a new set of exchange transaction log files.

After this hard part is done make sure you take backups on daily intervals. So after resetting your transaction log files you first thing to do is to take a full backup and then incremental or differentials on any days as per your backup schedules.

Thank you for all of the helpful input!

I enabled circular logging, dismounted and remounted the database.

That did clear all of the junk data from the drive as I'm back to 750+ free gigs.  

I'm not sure what I should be seeing in terms of available space in the screen cap below, maybe I'm not understanding that value as I assumed it would represent drive space available.  I did increase, but is still rather low...but maybe I'm confused as to what it should report.

I'm also not seeing any backup results from the PS command suggested, but backups have been successful over the weekend.


The script I ran to clean logs was from here https://www.alitajran.com/cleanup-logs-exchange-2013-2016-2019/

Should I disable circular logging at this point?

I'm not sure what I should be seeing in terms of available space in the screen cap below, maybe I'm not understanding that value as I assumed it would represent drive space available.

as i stated before, it is the white space in the database
when mail data is added and space is low, the edb file will automatically expand

The script I ran to clean logs was from here...

that will clean other logs like IIS and other exchange logs; as the article states, they are NOT database logs

Should I disable circular logging at this point?

if you have that much free space, then sure, disable circular logging
just watch your backups so that it doesn't exhaust the space again

I would like to add some more information as add-on to what Seth said.

circular logging has the disadvantage of allowing you to perform only full backups, rather than incremental or differential ones, because you can restore information only up to the last full backup.

So lets say to took full backup one week ago and your database goes down. Then if you took incremental or differential  for entire week you cannot restore those and you will only or can only restore full backup which is one week old. So in this way you loose 1 weeks data straight up.

So if you have enough space you need to always keep circular login disabled and rely on your backups and validate that after backups the transaction log files are getting purged.

Also the above command does not show an exchange backup was successful at any time. Verify your backups if they completed properly or were they failed.

Or just forget since now you have reset the log sequence and have dismounted and remounted which ultimately have created a new set of log files. I say go and take a full backup now and check the command status again once the backups completes.

Once you get the output that a full backup has been taken, then run some incremental or differentials after 1-2 days, then again run the command and verify that a new date and time is stamped on the backup set.

Thanks for the continued help, much appreciated!

So the backup issue is an odd one, maybe I don't have Veeam configured correctly?  I just switched to Veeam recently as I was using the native windows backup before.  I ran another full backup just before lunch, shows successful, but no change on the returned results in PS.


VSS Write list looks good:

But still no backups listed.

Do 3rd party backup solutions not work with Exchange?  This is all new to me...I assumed backups just took snapshots of the data and moved on, I didn't know the data was aware of whether it was being backed up or not and log files would grow exponentially.

Hold the phone...I did not have 'application aware processing' set on my backup job.

Let me run another full backup and report back.

Looks like I'm good to go!

That is perfect !

You can now disable the "circular mode" if not already configured (and redo a backup).

The main(and big) problem with the circular mode is that, in case of restoration,  you lose all modifications done between the time of the last backup, and the time of restore.

We generally estimate that logs of one database can take 10% of the size of this database. By security, as backup can failed sometime, it is a good advice to have 20% of free space for the logs.

Awesome, thank you so much, you all are life savers!!