Windows Server 2003 for Small Business Server SP2

Hi,
Can you tell us if anything changed?  Did you add patches/updates?  Was the server recently restarted?  Regardless, try restarting again, and if that does not help try a clean boot.  Have you tried scanning with an AV program?  Are there any errors in the event logs?  Have you tried SFC /scannow?  Run DCDiag or the SBS Best Practices Analyzer?   In Admin Tools - Services are all services marked automatic actually started?

1 - Because the system is out of date there are no patches/updates to apply. So the answer is no.
2 - Yes I restarted several times and did a complete shutdown
3 - I have TrendMicro Antivirus software, but did not try any scanning, I did however call tech support and they disable the Antivirus on the server and workstation to see if that helps, but it did not help.
4 - SFC /scannow - see attach file - Keep in mind that I'm troubleshooting with RDP, not at the Server locally
5 - DCDiag did not wok either
6 - I did not try SBS Best Practices Analyzer - Where do I find that
7 - Services - see attach file
8 - One thing I must point out is that I restore an image from Acronis Backup from a previous date before the problem started and everything work fine for a full day and the problem started again.
services.jpg
SFC.jpg

You can try disabled the Firewall and/or Antivirus and test again so can know if the Firewall is blobking this actions. If it works, you have to start creating rules in your Firewall/Antivirus.

Another thing, in some cases in my job I saw that the user have access to internet but not to other address or internal address not. So to resolve it, I disabled/enabled the network adapter and reboot.

Other thing is on CMD execute:

ipconfig /flushdns
ipconfig /registerdns

Select allOpen in new window

Does anything show in the Event Logs? I would also suggest to try removing Trend Micro as it can sometimes do weird things.

I disable it on both the serve and client side, but it did not help. Are you suggesting uninstalling completely?

Out of curiosity if your run \\ServerName\ShareName on the SBS itself, can you browse the share.  If that works it may not be related to a service, but would sound more like a firewall or A/V issue.  I agree with masnrock, many folks moved away from Trend Micro specifically on SBS years ago, when still supported, due to numerous issues.

For the record, or just a "heads up": some may recommend you "tinker" with the NIC such as uninstall, update drivers, reset TCP/IP stack.  SBS is VERY fussy about that and I have seen it result in a server rebuild, or restore from backup.  The IP is integrated in DNS, Exchange, SharePoint, and much more.  You must run the wizards if changes are made to networking.

Disabling isn't the same as uninstalling. Could be a problem of software components. Are you able to remove the client from the server, and not the console? (that would be ideal to avoid breaking the clients)

I will try uninstalling today after 5pm. and see if that helps. I also would like for you guys to check out the Event Viewer attachments.
Event-ID.jpg

Can I run the troubleshooting tools mention  SFC /scannow and DCDiag using RDP?

You should be able to do so

In theory, but you have to connect to the server with the console session to run SFC now, and it will probably ask for the install CD. To use console session:
mstsc  -v:servername  /admin

DCDiag shouldn't be a problem.

PS- on server 2003 if /admin doesn't work try  /console

Must say I have never run SFC on SBS.  I would be afraid it will replace current system files with the CD files which may be much older than current SP and patch files.  Doing so might require re-installing all service packs.  The order on SBS 2003 is very important, thus it could open a whole can of worms.

You should be able to run SFC and look at the recommendations, if any, then cancel.  I think you have been warned, SBS 2003 is unsupported and you could easily be in a failed server position when making changes.  Sounds like you have a tested backup though, so that takes a lot of the pressure off.  The difference with your 1985 Toyota is a company is not depending on it  :-)

try clearing the cache and restarting all the services one by one

You need to run sfc /scannow as an administrator ,so open a command prompt ,right click and run as admin.
As for your network connectivity,under to do list ,run the connect to the internet wizard again and see if that fixes things.

As for patches, M$ released a special patch for the wanna cry worm for the old stuff.
Download and install it.

http://www.catalog.update.microsoft.com/search.aspx?q=4012598

Interesting how many similar questions there are relating to suddenly being unable to access file shares on Server 2003.  All occurred over the weekend, and all seem to have RDP access of some sort to the site.  I never open port 3389 (I use TS gateway), but did for one client to a PC a while back.  Though that often gets hammered with failed attempts resulting in user lockouts the logs show the only recent attempts were over the weekend, and non stop.  I have since closed the port and denied RDP access.  The point is the time frame was interesting.  

This new Ransomware does attack SMB1, a file share protocol used on Server 2003, but the symptoms seem totally unrelated.

The only 2003 server I have with any clients, is SBS 2003 R2 and was patched with the SMB patch on the weekend has had no issues but there are no open ports on the firewall; i.e. no SMTP, RDP, or RWA.   This 2003 was stretched, time wise, to the limit as the company has known for some time they are shutting down the end of the May.  We are actually in the process of replacing all of our clients' SBS 2011.  

Just a point to ponder, file shares, server 2003, and RDP.   May not be ransomware, but perhaps a virus, hack, or similar.  Seems like a coincidence for numerous servers that have run without issue for years.

Have you run tools like TDDSkiller, Malwarebytes, HitmanPro ?

No I have not.

I also noticed slow Authentication to DC when user turn on their computer in at the beginning of the work day. About 5 minutes.

That is usually a DNS issue.  It happens when the client's NIC configuration has both the server and an ISP or router as an alternate.  The server MUST point only to itself, and the clients point only to the server, for DNS.  The ISP gets added as a forwarder in the server's DNS console.

As mentioned in your other question, much of this may be DNS related.

Every device in this environment has static IP addresses. See attachment
DC-ipconfig.jpg
OMCATRM-ipconfig.jpg

Get rid of that second DNS server on the DC.

The OMCATRM is fine but having 216.199.254.9 as an alternate on DC is a serious problem.  It should be deleted and run  ipconfig  /flushDNS

masnrock types faster....or less :-)

I try that already it did not help.

Regardless get rid of it.  #1 DNS mistake.

ISA server is not installed is it?  It was an optional add on with SBS 2003 premium, but has a lot of firewall options.

Have you rerun the connect to the internet wizard?

216.199.254.9 is my ISP Windstream.

ISA server has been removes years ago and replace with sonicwall

I have not rerun connect o internet wizard, because I wasn't sure if it will cause any problem.

I am attaching a file with detail description of when this all started. Please review and advice.

Thank you.
RPC-Error-Started.docx

>>"216.199.254.9 is my ISP Windstream."
As mentioned, that is a problem !!!!!!  When you try to resolve an internal name it often contacts the alternate first, or it responds firs.  If it does, it cannot resolve the internal DNS name.  When this happens you get delays tll it times out, or sometimes fails all together.

Add the ISP as a forwarder in the DNS management console.

The connect to the internet wizard should not caused problems unless you have "customized" SBS.

You have 3 open questions on the same issue making it difficult for us to track everything.  I believe in one of the others you said you restored from backup and it was OK for 24 hours.  If so that means you have a good tested backup so you should be OK to run the connect to the Internet wizard.

EE expects you to close a question before restarting a new relating to the same problem.

Sorry maybe my mistake. I wasn't sure how to close a question

The other two servers are just member servers ,right?
No DC or DNS?

Yes

Yew Restore was good for 24 hours.

I check the DNS management console and it's already been added as a forwarder

Good.  It just can't be in any NIC configration

6 - I did not try SBS Best Practices Analyzer - Where do I find that

https://support.microsoft.com/en-us/help/2673284/windows-sbs-best-practices-analyzer-bpa

Run it and post results.

Rob please list the steps in order that I need to follow and can I apply those steps via RDP?

Thank you.

Running the connect to the internet will probably kick you off the rdp session,so you need to run local.
The BPA analyzer should not.

Ok I will try at 5pm EST and keep you posted.

I am locally at the server now. I remove 216.199.254.9 from nic, did ipconfig /flushdns, ran internet wizard and reboot the server, but that did not fix the problem.

ran the sfc /scannow, but don't have sbs 2003 CD

Forgot to upload file
SFC-Scannow.jpg

After removing 216.199.254.9 from secondary DNS, I got the attach message from Eventvwr DNS server
DNS-Server-Eventvwr.jpg

Try running the sbs analyzer.

Attach is the sbs analyzer report
SBSBPA.SBS-2003-Scan.201705191702491.xml

You running this as a VM?

no. Sitting at the local DELL PowerEdge 2800 Server and ran the tool from the desktop.

Sorry I am out of the office and slow responding.  You asked; "Rob please list the steps in order that I need to follow and can I apply those steps via RDP?"
Which steps?  Removing the DNS server, which it sounds like you did.

There is no way those DNS errors relating to active directory are from removing 216.199.254.9.  You could get those errors if you removed 192.168.16.2

Might you have 2 network adapters on that server?  ISA required 2.  Perhaps post the results from the SBS by running   ipconfig  /all

From the bpa it does call out two network adapters being present.

<Value>192.168.16.2</Value>
                                                                                    <Rule Name="2IPs" Title="Multiple IP addresses assigned to the internal network adapter" Error="Warning" Query="count(($_/../Value | $_/../Instance)) &gt;1" Text="The internal network adapter of this server has two or more IP addresses assigned to it. You should remove all but one IP address." AlwaysEvaluate="True" Pass="False" />

I trust you ran the BPA before removing the 216.x.x.x address?  The BPA resultstates, as I have warned; "The DNS client is not configured to point only to the internal IP address of the server"

It also says it is running as a virtual server.  You may have issues with a virtual switch.
It shows 11 "networks" which means there have been 11 NICs or 11 reconfigurations, you may have a ghost NIC.
As pgm554 suggested, run the Connect to the Internet wizard which may fix these issues.  also as he said, best to run from the console.

It states disk space is very low.  If low enough some services may not start.

It is difficult to go through it all in XML format, you are best to look at the BPA results in the console and it will tell you the key issues and recommendations to fix.

Do a copy and past that looks like this
bpa.PNG

Rob Attach is the info you requested. I do have 2 adapters, but the other one is disable. I am only using 1 adapter which connect directly to my sonicwall.
ipconfig-all.jpg

pgm554 see attach file. There are 2 network adapter, but the other one is disable.
Public-NIC.jpg
Public-NIC2.jpg

Is it actually disabled or disconnected?  Ipconfig implies disabled, which is what it should be.

1 - Rob I ran the BPA after removing the 216.
2 - I do not have any virtual server. I only have 3 physical servers.
3 - See attach file for Server Management, is that the console you are talking about?
4 - I have 35GB free space.
5 - How do I look at the BPA report in the console?

disable and disconnected. There is no Ethernet cable connected to that adapter.

after you run the BPA there is an option to view report

pga554 all I have to do is copy and paste and just rename to bpa.png?

you can name it anything you want.

Attach is the BPA result from console. Let me know if I need to make any changes to registry.
BPA.jpg

BPA all issues attached. The reason I cannot send you the file in .png format is because I am remote into the server and cannot email you from that server. If you still need .png format, I can email to you on Monday, but if you need for me to check a particular item, then just let me know and I can make the change from my desktop.
BPA-all-issues.jpg

It mentions daylight saving time, does the server time and all connected devices match?  If off by more than 5 minutes you will have authentication issues and more.
TAsk offloading is more of a performance issue than a connection issue

Daylight savings time matches.

I have an idea, but need some assistance. I can do another restore from backup because I know for sure it works, but I need some type of monitoring tool in place to capture the point of failure when it does happen again. Any suggestions?

Here is what I can do:

1 - Restore  a full image backup like I did before
2 - Configure Monitor Filter within my Sonicwall
3 - Start capture traffic from one of my workstation to the DC
4 - End user of that workstation will notify me if problem occurs
5 - I will then stop capture and I should be able to view a report with the errors.

What do you guys think?

If it worked for 24 hours after the restore ,then it looks like something got updated.
I had an issue with AVG for Exchange that only affected sbs 2003 as it was the updated scan engine that broke it.
No issues with sbs 2011 or a any other machine.
Considering that support for 2k3 has ended ,you might have hit an update they didn't test.

1 - I am running Exchange 2003 on my DC which came with SBS 2003 Premium Edition, but my antivirus software is TrendMicro, are there any issue with that?
2 - There are no updates for SBS 2003 for years now.

I have seen Trend Micro cause a number of issues in random Windows servers. Sometimes a simple remove and reinstall works. Other times, removal and fixing other issues becomes necessary.

>There are no updates for SBS 2003 for years now.

Just a side note ,you can hack 2003 to look like XP POS which has security updates until 2019.
It does work,but use at your own risk.

http://www.zdnet.com/article/registry-hack-enables-continued-updates-for-windows-xp/

My point is that programs like AVG will update themselves regardless of OS and one of those updates can be the issue.

Just for the heck of it ,is there anything in the windows event logs?

Was the BPA-all-issues.jpg good enough or do you need a more detail report. I am not sure how to save it in another format. Please advice. I am going to the office now.

Only on the DNS Server. See attach file.
DNS.jpg

Hmmm...Event ID:      2004
Task Category: Resource Exhaustion Diagnosis Events

Sounds like you got a memory leak.

In task manger under the performance tab can you upload a screen shot of the current memory and CPU stats?
On a side note, is the server physical or virtual?

He already said it was physical.

1 - File Attached
2 - Physical
3 - Also guys when user first log into windows from their workstation it takes about 5 minutest to Authenticate to the DC.
Persormance-Tab.jpg

I will try another restore from backup today, but I need to have some type of tool running so I can capture the error when it happen again. What you guys recommend?

Could you please post screen shots of the Local Area Connections properties? Want to see the clients and protocols that are present.

See attach file.
LAN-properties1.jpg
Lan-porperties2.jpg

Let's analyze sections of your issues...
1) How is your hard drive partitioned? Is OS in a separate partition from everything else? If so, how much space is left where the OS is?
2) Have you tried correcting the .Net issues?
3) Have you tried disabling Task Offload?

1 - Raid1 - OS - Free space = 36GB, Raid5 - Data files - free space 36GB
2 - No did not tried correcting the .Net issues, don't know where to start
3 - Did not disable task offload

I'd start with disabling task offload. Check out properties for your NIC cards (will most likely be in the Advanced tab)

Like I said before I did a full image restore from previous backup and the system was working fine about 24 hours, then the problem occur again. I would like to do another restore, but I need recommendation to have some type of sniffing tool in place to capture the error next time around. Any recommendations will help.

I do remember that portion, but something has to be causing the error at a certain point after the restore. You could use Wireshark and do a packet capture. Obviously we won't be sure what to tell you to look for, other than something out of the norm.

I don't imagine there are any tools that will say "ah hah, here is the related error".  There are tools that will alert if connectivity is lost but that is just a ping test and you say that works.  Basically you need to wait for a problem, note the time,  and review the event viewer for all errors.  You can filter by warnings and errors.  Even so, those errors should be there now.  There are tools like https://www.spiceworks.com/  that allow central monitoring and you can set up some alerts.  I still think you should run the connect to the internet wizard as pgm554 recommended.  If you are about to do a restore there is no risk.  24 hours of Wireshark would be a lot of data, and you would need to know what you are looking for.

Attach is what I get when I go to advance tab
NIC-Advance-Tab.jpg

I  did ran connect to the internet wizard.

That error just indicates RRAS is enabled

It would be interesting to do a restore and put in a false gateway  address, on the SBS, for 48 hours.  That would block any updates, Windows or A/V and should even block outside access/hacks.  You can still access from the LAN.  And you could still remote in to the TS. The problem with that would be no device on the LAN would have internet access due to DNS forwarders not working.

Was the BPA report I ran not good enough. If not please tell me what I'm doing wrong so I can rerun and send to you.

The problem with a packet capture is you need a switch that can mirror the ports so you can do an analysis.
Some have the ability ,others don't.

Most of your critical errors point to Active Directory, which is most often caused by DNS but from what we have seen, after your changes, it is OK.  Have a look in the event viewer under "Applications and Service Logs", in the "Directory Services" and "DNS server" logs for any red Errors since the rebuild.

Sorry, on 2003 there is no  "Applications and Service Logs" group.  "Directory Services" and "DNS server" logs are just at the root of event viewer.

I check the Directory Services and DNS logs no red errors.

>I check the Directory Services and DNS logs no red errors.

No ,at one point it had :
Event ID:      2004
Task Category: Resource Exhaustion Diagnosis Events

Was it 2004 or 4004?  I couldn't find former.

4004, but they say that's normal when you reboot the server.

yes

After I uninstall Trendmicro completely from serve and workstation and reboot. Attach is the following message from eventvwr.
eventvwr-after-Trendmicro-Unstall.jpg

Those are all the same as before.  any changes to connectivity?

No. Struggling right now.

I did another restore from backup and everything was working fine again, until another 17 hours, the problem came back. However; I cleared the logs in event viewer after I did the restore. There were no errors, just a warning message which I attach, could this be the problem?
Event-ID-1706.jpg

I doubt that has any bearing on your primary issue.  You say 17 hours but there is only 6 hours in the event viewer.  What time of day was it that it stopped working?  
Check all scheduled tasks for anything around that time like AV updates, Windows Updates, defragging, etc.

Check your SMTP services and make sure nothing looks odd: http://support.microsoft.com/kb/914137

However, I doubt this is your main issue, just another one that happens to exist.

Is your server able to ping and communicate with other systems without issue?

Yes, I can ping to and from all computers and workstations within my network. My issue is I cannot access any resources from my DC (shares, mapped drives, network printer, etc.). Vise versa it works file. I can access any servers and workstation from my DC, but I cannot access anything to my DC. So it works one only.

Have you checked that NetBIOS over TCP/IP is enabled?

Yes it's enabled.

With the exception of checking whether SMBv1 is enabled, everything seems to so far be in a normal (or at least tolerable) state. Could you please verify that SMBv1 isn't disabled?

Wonder if it is getting the SMB Ransomeware patch after X hours and causing the problem.
As mentioned before; is it possible to do a rebuild and disconnect from network or give it a false gateway address and see if it continues to work.  If you are not using it for Exchange false gateway would allow users on the network to use the server but block it from Internet traffic such as Windows or AV updates.  However it would block public DNS queries and your remote access.  For remote access you could access a PC and the server from it.

What's SMBv1?

@Rob - I don't think Microsoft patched the Small Business Servers.

SMB = Server Message Block, which is one of several Windows uses for communicating with other systems on a network.

Here's an article to check whether it is enabled or not, and how to correct: https://support.microsoft.com/en-us/help/2696547/how-to-enable-and-disable-smbv1-smbv2-and-smbv3-in-windows-and-windows-server

@masnrock:  

Microsoft did release an "out of band" patch for 2003 on the 14th, which I applied to an SBS 2003 (actually decommissioning that server tonight).  However at that time it was not being automatically deployed.  You had to download and install.  I don't know if that changed and might be installed by WSUS now.

Regardless, as you suggested before, it sounds like something is updating, or a scheduled task causing the problem.  Though SMB is an excellent item to check, it is unlikely it would disable itself after 18 hours.   It may be a result of the real problem.

Nope they have a patch.
I installed one on SBS2003 recently.

I checked the entry below, there is no entry for SMB1 under Parameters

To enable or disable SMBv1 on the SMB server, configure the following registry key:
Registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\ParametersRegistry entry: SMB1
 REG_DWORD: 0 = Disabled
 REG_DWORD: 1 = Enabled
 Default: 1 = Enabled

I see you're running Acronis.
In my many years of IT,the two biggest PITA's were virus software and backup software.
Remove both and see what happens.

Was anything like Client for Microsoft Networks unchecked when you looked at the properties for your network connection?

network properties  are all ok.

I can't really see it being a setting, A/V, or backup software as everything works for 17 to 48 hours after a restore from backup.  Something is changing.

Simple test,just disconnect from internet and if the issue doesn't come back while disconnected,but comes back when connected,you got something updating from the internet.

Are you using IPSec at all?

@pgm554 "just disconnect from internet "
As suggested twice earlier, however I mentioned alternatively, assigning an incorrect gateway may have less impact on end users as disconnecting from Internet, with 1 NIC, means disconnecting from the network as well.

An alternative would be to set it up as a test standalone vm and do the same thing.

Moreover,do a restore and note the builds of the virus scan engine and backup up software.
If the build engine changes ,then you know you have an auto update issue.

As I pointed out ,I had an issue with AVG that ONLY affected sbs 2003.
The problem I had was the signature updates included the new scan engine which broke Exchange and it would take a couple of days for the server to fail.

A VM is a good idea.  I agree SBS and various A/V apps have a long history of problems, assuming it's related to an update.

I disjoin and rejoin an xp workstation from the domain and receive the following attached.
Remove-Workstation-from-DC.jpg

If you cannot access any services on the DC you won't be able to join the domain, though that sounds like a DNS issue.  Does the PC point ONLY to the server for DNS?  You cannot have an ISP or router, even as an alternate/second.

Yes  DC point only to the server for DNS.

You need to solve the primary issue first.  The only option I can see is rebuild again and disconnect from the Internet as I and others have stated.

I've decided to think outside the box and check he Application Event ID: on one of my win 7 workstation and my win server 2008. Attach is the file, could this be my problem?
Application-Event-ID-1001.jpg

First, please read my post on effective screenshots:
https://www.experts-exchange.com/articles/29715/Effective-Screen-Shots.html
(I'm specifically referring to embedding).

Second, you need to review the issues found with DCDIAG /C /E /V and REPADMIN /SHOWREPL

Problem was not resolved. Had to purchase new 2016 Servers and recreate my network