Just want to share experience of recent issues that we have faced in our exchange infrastructure , as a team we have handled it very quickly. It was a normal support day but at about 11:00 AM CST , exchange team started getting critical alerts that mail queue on hub servers have increased & after a moment all the database got dismounted on mail box servers.
Information store service was crashing, at first we have tried to start the Information store service on one of the server for which we have received alerts but after 10 mins it crashed again ,in the mean while we got alerts from other servers too.
I have made a guess based on my experinnce that the probale reason can be ANTIVIRUS.
We have disabled the Group shield on exchange hub transport servers, process is pretty simple.
Mcafee has provided a script Disable-Agents.ps1 that is present in C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\E2007 Agents –> execute this script & if transport service is not stopping just kill thru task manager.(don’t close the script window)
On Mailbox servers we have stopped group shield service.
These two steps given us temporary fix, meanwhile we logged a call wuth Mcafee & came to know that the DAT they have released (6682) is bad dat & it has affected other customers too.
We have then rolled back the DAT to older version so that we have Antivirus protection.
As group shield has been stopped on Mailbox servers & agents has been disabled on Hub servers so we followed below steps to re-enable protection after coordination with Mcafee.
1. Open regedt 32
2. HKEY_LOCAL_MACHINE\SOFTWARE\McAfee\GroupShield for Exchange\SystemState
4. 6682.0000-1 –> Change it to 6681.0000-1
5. got to C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\DATs & delete the 6682.0000-1(bad dat)
6. Now use Enable-Agents.ps1 in C:\Program Files (x86)\McAfee\GroupShield for Exchange\bin\E2007 Agents (HUB servers)
7. On mailbox servers sytart the Group Shiled service.
Also we have disabled th scheduled update of DAT until Mcafee has relaesed as NEW DAT.
Note:- Use Get-Transportagent command on HUB transport servers to query the status of Mcafee Aagents.