David Ackerman
asked on
Vendor Issue and Credit Negotiations
I have a festering issue with the owners of my company and our I.T. consulting/hosting vendor. Our Exchange Database corrupted On the evening of Tuesday, 1/31 and took email down. Email was offline for one business day but send/receive was available by Thursday. However we had to restore mailboxes incrementally.
The database is huge... 1.4 terabytes so obviously the restore would take some time...
Restoring back to 1/1/17 (fully restored by Monday), back to 10/1/16 (by end of week), and balance by beginning of following week.
My controller is demanding a full month's credit from our vendor for all services provided (email, hosting, support, maintenance, etc)
Total nut is $25k. Email hosting svcs represent about 20% of monthly fee.
I've spoken with other people regarding this and the consensus is this demand is just stupid. But the controller had sold the owners on the "major disruption" this caused and how we lost revenue.
Now, we are definitely an email dependent business, but the question is... Can the vendor be held that responsible for a database corruption? Are my owners/controller being close to reasonable?
Any thoughts would be appreciated.
The database is huge... 1.4 terabytes so obviously the restore would take some time...
Restoring back to 1/1/17 (fully restored by Monday), back to 10/1/16 (by end of week), and balance by beginning of following week.
My controller is demanding a full month's credit from our vendor for all services provided (email, hosting, support, maintenance, etc)
Total nut is $25k. Email hosting svcs represent about 20% of monthly fee.
I've spoken with other people regarding this and the consensus is this demand is just stupid. But the controller had sold the owners on the "major disruption" this caused and how we lost revenue.
Now, we are definitely an email dependent business, but the question is... Can the vendor be held that responsible for a database corruption? Are my owners/controller being close to reasonable?
Any thoughts would be appreciated.
ASKER
Ironically, we were/are in the midst of a migration to Outlook 365 when this happened...
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
As Lee state above this is down to the original contract you have in place with the MSP/Support provider. If i am reading the above correct you have a contract for $25,000 per month with your provider. If this is the case i would have hoped that you have a BIA (Business Impact Analysis) and risk profile that is presented back to the business on a regular basis for sign off or approval (Governance is key and this is a good example of why not only Service providers should use it but also why you should build one internally to ensure the business is aware of all risks, exchange risks are very obvious)
If you have the above available this should highlight the risks to the business and any remediation time/costs if the business wants to mitigate the risk or the business accepts liability in the event failure occurs. The only thing i would note at this point is if you were charged for the recovery of the exchange as the above would no doubt have taken in excess of 20 engineering hours to complete, you might want to ensure that cost is also not forwarded on to the business (if a risk is accepted the client might accept remediation costs for not investing in a redundant solutions to begin with).
If you have the above available this should highlight the risks to the business and any remediation time/costs if the business wants to mitigate the risk or the business accepts liability in the event failure occurs. The only thing i would note at this point is if you were charged for the recovery of the exchange as the above would no doubt have taken in excess of 20 engineering hours to complete, you might want to ensure that cost is also not forwarded on to the business (if a risk is accepted the client might accept remediation costs for not investing in a redundant solutions to begin with).
ASKER
Sadly, I agree with everything you say. I appreciate your feedback.
I would say that overall, they have always done right by us and have provided an entire Iaas platform for us for what I would view as a fair price.
Fully virtualized infrastructure, hosting , Citrix Desktops, backup/maintenance/monitor
That's the lions share of our IT budget on revenue of $50-60 million (.05%!).
Maybe I'm naive but I don't think I'm going to do much better than that...
I'm just going to have to deal with the fact that this stance is now based on emotion and not a reasonable and quantifiable position.
I would say that overall, they have always done right by us and have provided an entire Iaas platform for us for what I would view as a fair price.
Fully virtualized infrastructure, hosting , Citrix Desktops, backup/maintenance/monitor
That's the lions share of our IT budget on revenue of $50-60 million (.05%!).
Maybe I'm naive but I don't think I'm going to do much better than that...
I'm just going to have to deal with the fact that this stance is now based on emotion and not a reasonable and quantifiable position.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
LBTechSol to your point, no we did not receive any kind of BIA. The issue with Exchange being a risk point had been addressed in a somewhat less formal fashion and was being addressed in our migration to Outlook 365 when this unfortunately occurred.
The point is well taken as i suppose my faith all points of exposure being completely analyzed and addressed to that level was misplaced.
The point is well taken as i suppose my faith all points of exposure being completely analyzed and addressed to that level was misplaced.
ASKER
Agreed LBTechSol,
As of now, my vendor not only covered all expense of the recovery under our standing service agreement (and I can confirm from having worked with them through the whole process that it was a significant amount of man hours), but are willing to build a redundant server configuration at no charge or issue a credit for the value of that project ($5-6k).
If (and as of this moment it's a big if...) my owners were willing to strike this deal, I lean toward taking the risk for the next two months until our Outlook 365 migration is complete since it would render that interim server configuration moot and apply the credit to our current balance.
At the end of they day, the decision is going to be theirs, I am just faced with the fact that the only satisfaction acceptable to them is outside the realm of reasonable negotiation.
And then we will be looking at moving our infrastructure to an entirely new vendor/facility...
And I don't want to think about how much disruption that will cause...
As of now, my vendor not only covered all expense of the recovery under our standing service agreement (and I can confirm from having worked with them through the whole process that it was a significant amount of man hours), but are willing to build a redundant server configuration at no charge or issue a credit for the value of that project ($5-6k).
If (and as of this moment it's a big if...) my owners were willing to strike this deal, I lean toward taking the risk for the next two months until our Outlook 365 migration is complete since it would render that interim server configuration moot and apply the credit to our current balance.
At the end of they day, the decision is going to be theirs, I am just faced with the fact that the only satisfaction acceptable to them is outside the realm of reasonable negotiation.
And then we will be looking at moving our infrastructure to an entirely new vendor/facility...
And I don't want to think about how much disruption that will cause...
Use this "Issue" as a reason to request documentation. Let the service provider know that "The Business" has requested the information to ensure this does not happen again, let them know you have to build a BIA for all critical services hence the need for information. If you are looking to move to another supplier make sure you get everything you can from the existing company before they think they will loose the business as this might be when they become distant and slow with information. Next take your time when looking for another Vendor, the sales people always promise the world Talk to the Tech guys, visit the offices and complete full due diligence as some times the grass is not always greener on the other side.
The issue comes down to what your SLAs included concerning eMail in the provider contract and what requirements you listed when the contract was bid.
I would not casually discuss this stuff with either party. Put together a brief easy to understand presentation for your management. Include:
How and why of the outage.
The agreed upon SLA with provider.
Plans to prevent/mitigate future outages
Be mindful of your company politics. If the vendor clearly did not meet agreed upon requirements (you need a working relationship to accomplish anything going forward -- even throwing them out). If your management stupid decisions contributed don't be too blunt about it. You never know whom up the chain will see or hear this and some may have been involved. Provide brief honest information your audience can easily understand. This is not the place to show off your in depth knowledge! Rather its a list of facts your superiors can digest (think 4th grade reading level and as little technological crap as possible). You can always answer questions regarding details. These a smart people they can figure things out.
I would not casually discuss this stuff with either party. Put together a brief easy to understand presentation for your management. Include:
How and why of the outage.
If you have not already done so, convene a formal post mortem, to determine what happened (root cause), why it happened (mistake? Hardware? Outside issue?) and how to prevent this specific event down the road. Boil it down to digestable size.
The agreed upon SLA with provider.
Include how it was or was not met. Cover how this eMail SLA was determined in the first place. Vendor suggestions, Cost issues and related decisions. Be careful here see my politics note below.
Plans to prevent/mitigate future outages
Sounds like you already knew the current eMail infrastructure was vulnerable, describe how current plans will beef up availability in general as well as this specific situation. Include costs, changes in cost and justification.
Be mindful of your company politics. If the vendor clearly did not meet agreed upon requirements (you need a working relationship to accomplish anything going forward -- even throwing them out). If your management stupid decisions contributed don't be too blunt about it. You never know whom up the chain will see or hear this and some may have been involved. Provide brief honest information your audience can easily understand. This is not the place to show off your in depth knowledge! Rather its a list of facts your superiors can digest (think 4th grade reading level and as little technological crap as possible). You can always answer questions regarding details. These a smart people they can figure things out.
No comment has been added to this question in more than 21 days, so it is now classified as abandoned.
I have recommended this question be closed as follows:
Split:
-- LBTechSol (https:#a42051386)
-- Lee W MVP (https:#a42051334)
If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.
jackieman
Experts-Exchange Cleanup Volunteer
I have recommended this question be closed as follows:
Split:
-- LBTechSol (https:#a42051386)
-- Lee W MVP (https:#a42051334)
If you feel this question should be closed differently, post an objection and the moderators will review all objections and close it as they feel fit. If no one objects, this question will be closed automatically the way described above.
jackieman
Experts-Exchange Cleanup Volunteer
At the end of the day it's a legal matter.
We don't know exactly what happened, why, and frankly, anything you tell us is still only one side of the story.