Need to remove CPU Miner on exchange server asap

Alon Tabak
Alon Tabak used Ask the Experts™
on
CPU Minter Cryptocurrency on Exchange 2016 server, tried "everything" to remove and cannot find the culprit - need assistance please!
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Ganesh AnandLead Technical Consultant

Commented:
It appears your PC is been used for bitcoin by some hacker, because you have very easy password and open many ports from internet. You should not allow any ports other than 443 and 80 (inbound). If you have any SMTP gateway port 25 should be allowed to secure. If suppose you also use POP3 and IMAP then respective secured port should be allowed.

First reset the local administrator account with high complex password note down.
Disable or reset any other local accounts.
Disable RDP port from external and try to use secure VPN access for RDP and any other port.
Remove the miner from the Program and Features, but you will see some of the miner.exe will be on the process. You need to right click the process and open file location and kill the process and remove those source files and emtpy from the recycle bin.
If you couldnt succeeed, restart the server in safe mode and remove the process, source and cleanup the necessary files. Remove %temp% files and folders from the c:\windows\temp as well.
Uninstall the existing antivirus and install the antivirus.

Let me know if this solved your issue.

Author

Commented:
Hi, I want to thank you very much for taking the time to answer my question.
I actually use Cisco Sourcefire will rules to protect, and no RDP from outside and only port 443 for OWA (which is disabled for most users, Outlook anywhere (phones), 443 is also used for vpn.  I always try to get my ciphers ssl up to date, but people are sophisticated out there.
My sourcefire says i have a crypto miner, but I cannot find it anywhere at all - i looked in services, resource monitors, task schedulers - i'm ready to give up and call msft - which i'm not even sure they can assist with.

I'll look at the windows temp again.

Just want to mention, if anyone has a good grip on these gpu type malware i'm happy to PPM to fix.

thanks again!
Ganesh AnandLead Technical Consultant

Commented:
Normally this crypto miner will uses easy password to compromise the system, for example passwords like simple password, companyname@123 kind of easier predictable passwords are more easier and judgeable. You should not use simple passwords in order to avoid this kind of issues.

Did you checked the miner.exe process running on the exchange server and did you check if it is in the programs and features?

Normally this miner will get installed under c:\programdata\

Try malwarebytes, or some good AV which can remove miner malware.

Read these references which might help you: https://www.trendmicro.com/vinfo/us/threat-encyclopedia/malware/worm_coinmine.b
https://www.bleepingcomputer.com/virus-removal/remove-trojan.bitcoinminer-miner-infection

https://answers.microsoft.com/en-us/protect/forum/all/bitcoin-mining-virus-in-mspoolvexe/20c9f937-5bb7-4db2-95c3-905bea01c31c
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
This is the exact error I get on my SourceFire:        PUA-OTHER CPUMiner-Multi cryptocurrency mining pool connection attempt (1:45550:1)

I spoke with an expert at Cisco and they said it's the real deal.   I used MalwareBytes, SpyHunter, Symantec Endpoint and nothing - all clean.

I installed Windows Process Explorer to try and find the culprit.

I looked in all c drive temp, etc  All files unhidden

I spoke with MAS who gave a good suggestion to redo the C drive, then point all exchange services to the "new" machine.

Not my forte (little scary) but i have some backups thank goodness.

I'm open to cleaning it first - as this is the easist method.

So far, this has been a tough one!

Author

Commented:
Also the server itself is not requesting (outbound or inbound) connections to the "mother ship" crypto miner.  It's actually sitting on this server getting replicated to another site using a debian linux box (for replication) and each time it finds this packet (with the malware) it resets.

this is definitely a tough one!
Ganesh AnandLead Technical Consultant

Commented:
Did you find any process running related with the miner, sometimes it has similar names like svchosts instead of svchost.exe. You need to verify all the processes and find the source location which you can find the root cause.
With what have you scanned the computer?  I'd give Malwarebytes Anti-Malware and SuperAntiSpyware a try.

The most common EE response you'll get is to wipe and restore from a known good backup.  That's the "safest" solution.

Author

Commented:
I've tried Malwarebytes and SpyHunter and Symantec Endpoint but all found nothing - I feel SpyHunter may have been the most thorough, but still nothing.  I spoke with MAS (one of the tech's on this site) and he suggested (a recent c drive restore from backup) - which i'm embarrassed to say this issue has been going on for over a month (so that really will not work without AD issues).  The other option is to install a new C drive and mount the other drives to this "new" old VM.  I'm not super comfortable with that option but have a snapshot/backup just in case.

The strangest part of all this is that I cannot find the culprit - and @Ganesh - I looked at the processes several times to no avail.  The sourcefire tech who was fairly senior mentioned that sometimes these bots appear and then erase themselves.  This is probably the hardest single issue I've ever faced!  sweating!
Brian MurphySenior Information Technology Consultant

Commented:
Sounds like you need to look internal?  

As in an internal, possibly admin, employee.

Malwarebytes, SpyHunter and Symantec detect infected Mining packages.  Not legit ones.

It is not uncommon to use CPU or GPU to mine Bitcoin, Zcoin or the hundreds of others available.

It just seems odd that they would just target the Exchange server.  It sounds like you have already verified with your Firewall support this is not a false alarm.

If you simply open CMD prompt as Administrator and type "tasklist", do you see anything like cminer or miner.exe?  These programs do not require Install, merely transfer.  But the transfer would require some type of vulnerability to a simple copy process.

However, if you see nothing, I would be concerned regarding a relatively newer vulnerability called PowerGhost Malware.  It has been spreading across corporate networks infecting both servers and workstations to illegally mining the crypt-currency and Perform DDoS Attacks.

This type of attack does not require any files on the remote system.

I've not actually heard of it being in the US, but....Windows servers, have been infected by a Monero miner known as Smominru, according to researchers at Proofpoint. It spreads using the EternalBlue exploit (CVE-2017-0144) which targeted the SMBv1 protocol.

When it comes to detecting Cryptocurrency mining, you need to be looking at DNS client traffic as well as IRC communications.

It is possible to install a Ghost miner on one "server", in this case, and bounce that traffic off another workstation.

Similar to aggregated VPN's using Virtual Machines.  For example, is your Exchange Server running Hyper-V?  Perhaps the miner is running in a VHDX?  All your AV/Malware installs won't necessarily detect this?

Let me know if you want to discuss further?

Author

Commented:
Hi Brian, this is really great information!  The way I noticed the CPU Minter was via SourceFire only.  I have one person that lives in Puerto Rico and involved in bitcoin, but when I check his PC it always comes up "clean" - never thought that it could be undetected.  That said, the reason I say it's exchange is because in my Vmware environment (3 physical hosts, and Zerto Virtual replication appliances) no matter where I move the Exchange server (migrate) this PUA-OTHER CPUMiner-Multi cryptocurrency mining pool connection attempt (1:45550:1) follows me around.
I thought about a fresh install and somehow migrating the drives to the fresh install - but one fear is of course in case this is larger than just that and of course not being an exchange expert, I wanted to make sure I did this properly (as it's in prod mode).  That said, I would love to speak - I have a US landline or skype - "aloniscalling".  Much appreciated!  thanks
Brian MurphySenior Information Technology Consultant

Commented:
I certainly don't mind speaking with you but for the greater good I would state first that, it won't be that simple.

First thing you need is to mirror that port(s) at the switch(s) where your 3 physical host reside.  Not the virtual, I'm talking the actual physical port back to the switch must be mirrored.  They may very well be split between a core switch somewhere and cannot take any chances if you are using a software layer port aggregate.  Whatever ports those hosts can pass traffic on the physical switch ports must be mirrored.  Many things can be done to redirect that traffic. All that matters is the physical ports that the 3 hosts have access too....all of them.  If traffic is coming from those hosts, we can find it.  

Then, we use a combination of network traffic analysis and IDS to provide visibility, achieved multiple ways.

I'm not going to "tout" any specific product so I'll start with what you need to achieve out of a product.

Even then, I'm making assumptions here being the traffic can certainly be "redirected" or even "encrypted", and there is more.

Me being on the phone, is not going to cut it.  I would have to be there in person or be on the Network - if and when these more simple techniques don't work.

I would start simple....

There are not that many miners that still focus on CPU but GPU.  Also, look at the person you know in Crypto?  Did they make a mistake?  Did they download a Trojan?  Did that inadvertently infect your Exchange server?  Because, honestly, it is a bit of an anomaly.   If it was really smart, it would have attacked your underlying Linux host (aka VMWare) and used all the CPU's.  And, if it really uses CPU.... not GPU.... that narrows the gap.

Author

Commented:
The servers are in a Collo Site in NYC so I have to get there and start the process.  I was also thinking that updating the Hardware Firmware on the server equipment maybe helpful as well?!?  That said, I was able to "corner" the issue using SourceFire, as I don't see any connection outbound nor inbound to any "mothership" miner, but I still want to get rid of it on the exchange server.  Many suggested changing passwords, setting up a new server and migrating, which may be a possibility starting asap (before the weekend is up).  You seem like an expert at this issue so perhaps doing the easier updates, and migration will be the first step - see if I still get errors on the Firewall, then if nothing helps go for the onsite as you suggest.  I'm happy to pay for your time if you are in the great NYC metro area.  Only issue is I don't want completely lose my "mind" if we can squash it easily.  The only port exposed to the outside world is my 443 on exchange, so it makes sense that exchange has this issue.  I do have a firewall that protects but perhaps my ciphers are too weak.  

I apprecaite all of your time in either case, and hope we can work something out here to fix this issue - hopefully not needing to go "crazy" doing it.  BUT open to all suggestions as well.
reinstalling is probably faster and safer

that said, if you do allow the connection, process explorer will reveal the program that is connected. obviously other tool such as personal firewalls will be able to detect the connection attempt and corresponding process without allowing the connection in the first place. most likely, you are facing a worm that is buried inside other programs. possibly exchange itself.

do not expect a microsoft server accessible from the internet to stay unhacked

do not expect any msft machine that is part of a domain with regular users that will most likely get viruses to stay unhacked either
Can you see the miner running from Task manager, process tab? Should show as an unfamiliar program using a heap of CPU. If so, right click, open file location, and you should see the executable. Kill the task, delete the executable, then create a new file of the same name, with no contents, and make it read only, with no NTFS permissions. (to stop it from being replaced).

That should at least get you up and running while you patch any security holes and get some antimalware app to eradicate it properly.

Sometimes these things can be VERY difficult to locate and kill, hence the suggestion about recovering data and rebuilding the whole thing from the beginning. Sometimes that is the easiest and most certain way.

Author

Commented:
So the strange thing about this Miner (again based on SourceFire telling me it's there) is that It's not running in the Task mgr or process tab (unless it's being masked somewhere).  I'm setting up a replacement Exchange server but still monitoring the local firewall logs on the old server and so far nothing.  I assume the miners are not the one's that encrypt your files, etc - that would stink.  Last question - after this ordeal is squashed, how do you not expose 443 to the internet but rather through a proxy?  I know Kemp makes a product.  Thanks again for your replies btw - i do appreciate!
most likely, you are facing a worm that is buried inside other programs. possibly exchange itself.

most likely that's why you cannot see it in the task manager

how do you not expose 443 to the internet but rather through a proxy

you setup a reverse proxy : lighttpd, nginx, apache, squid, haproxy .... most web servers have a builtin reverse proxy feature. some do relatively exhaustive checks, others do barely more than a tcp proxy. and in that case the proxy usually bares the SSL terminations or is able to perform on the fly ssl decryption

Author

Commented:
@skullnobrains - I'll definitely check this out, i was thinking a separate physical linux box off my immediate system.
seems good to me. the box does not need to be physical as long as you don't open an extra security hole by using a virtual one.
To resolve - After setting up a new Mail server, and migrating mailboxes - it appears that the malware is coming from the mailstore itself.

I'm working with Symantec for Exchange to remove - found multiple viruses, but still haven't found the culprit that Cisco is finding.  Hope that this resolves at some point soon - thanks again everyone for all of your input, much appreciated!
it seems likely the mailstore would contain viruses.
but finding the same worm after migration does not necessarily means it is located in the mailstore.

you may want to consider other possibilities including a false detection or more likely this miner is installed by some other virus.
actually the most likely scenario is a virus capable of hopping from one host to the next in the same domain/network that hopped onto your new machine as soon as you joined the domain, and that malware is somehow responsible for installing the miner. and additionally that malware would already be present on many/all of the domain's hosts.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial