Issues at Office365 (Resolved) Issue / Performance
30 days

After speaking with O365 support we've learned about a new connection filtering policy in test. In order to override the default connection limits, you must override the default policy depending on your mail flow volume

  1. Login to Exchange 365 Admin Panel
  2. Select protection > connection filter
  3. Edit the Default policy
  4. Select connection filtering
  5. Add two new entries, one for each of our delivery range
    65.99.255.0/24
    206.125.40.0/24
  6. Save your policy


10/01/2020 17:17 PM 30 days

Customers on O365 will have to raise a case with Microsoft detailing the following error rejection codes


  • Deferred: 451 4.7.500 Server busy. Please try again later from [65.99.255.116]. (S77719) [CY1NAM02FT049.eop-nam02.prod.protection.outlook.com]
  • Deferred: 451 4.7.500 Server busy. Please try again later from [206.125.40.249]. (S77719) [VE1EUR03FT052.eop-EUR03.prod.protection.outlook.com]


Microsoft has started to throttle deliveries from ExchangeDefender with no rhyme or reason. We've opened up a case with Microsoft, but we urge partners to do the same.

10/01/2020 16:56 PM 30 days

We are again seeing random delivery failures and server busy errors while delivering mail to Office365. It appears to be affecting multiple clients.

The issue has been reported to Microsoft and we hope they sort it out quicky (The error they are reporting is  (Deferred: 451 4.7.500 Server busy. Please try again later from [65.99.255.116]. (S77719) [CO1NAM04FT)

In the meantime, this is why you have LiveArchive - you can continue to send and receive mail and access all the inbound mail that may be backed up. You can access LiveArchive at https://livearchive.exchangedefender.com

Email processing delays in Los Angeles (Resolved) Issue / Performance
19 days

We have seen slower processing times on two of our clusters in Los Angeles, inbound13 and inbound16 starting this morning around 9-9:15AM. Our standard diagnostics did not turn up any obvious issues so these systems have been pulled from production so that investigation can continue without delaying mail delivery.

We have removed those clusters from inbound mail routing and have moved all messages to another cluster to process. We expect that all the messages from i13 and i16 will be processed and delivered in the next 5-10 minutes (11:10-11:15 AM EST).

Investigating Exchange resource errors (Resolved) Investigating / Notice
7 days

The issue with Outlook Web Access has also been addressed now. 

We will continue to monitor and review the perfromance but we have confirmed that all services are functional and there are no accounts reporting errors or login timeouts.

08/24/2020 11:36 AM 7 days

The issue regarding mail flow has been addressed, all mail that has been accepted/delayed since approximately midnight (when the issue started) has been dequeued from ExchangeDefender and delivered to Exchange.

We are still working on the other reported issue related to Outlook Web Access logins. Please use LiveArchive (https://webmail.livearchive.us/) for the moment.

08/24/2020 11:07 AM 7 days

We are investigating reported errors in email delivery and OWA login that started earlier this morning. We will keep this post up to date as services are recovered back to normal.

Delivery delays Exchange 2016 (Resolved) Issue / Performance
7 days

Between 4:10PM and 5:01 PM we noticed delivery delays to and from Exchange 2016. Upon investigation, the delivery delays occurred at the Edge network which is responsible for message coming into or leaving Exchange. At 5:01 PM we were able to rectify the issue and begun to process queued messages. 

outbound rejection emails (Resolved) Investigating / Notice
7 days

A subset of outgoing messages may have responded with "host not found" between 11:30AM and 11:32AM

While developing new changes on an outbound node, we noticed the load balancer did not pull the testing server out of rotation and temporarily allowed mail to relay through to the test server. Unfortunately, the outbound test node wasn't fully configured to handle routing from customers and responded with the following error


DSN: Host unknown (Name server: 127.0.0.1: host not found)


The test node was quickly removed from the network and we confirmed it was no longer in rotation before continuing development. 



Critical issue solved on outbound3 (Resolved) Issue / Performance
1 days

At approximately 11AM EST, one of the ExchangeDefender outbound network endpoints (outbound3.exchangedefender.com) was brought into rotation without all the relay ACLs authorized. The result is the message below for some clients/domains whose policies did not load properly. The error they received is a 500 Relaying Denied, and it's been fixed/addressed. 

Your message wasn't delivered due to a permission or security issue. It may have been rejected by a moderator, the address may only accept e-mail from certain senders, or another restriction may be preventing delivery.

The following organization rejected your message: outbound3.exchangedefender.com.

If you received this error message, please go to your Sent Items and resend the message. 

Exchange2016 OWA and Outlook connectivity issues (Resolved) Issue / Performance
2 days

The issue with Exchange has been addressed. We started a major project last Thursday involving the entire Exchange team to address series of expoits that have been documented in the wild and could pose threat: https://swarm.ptsecurity.com/attacking-ms-exchange-web-interfaces/

The actual issue that caused a problem was unrelated, in one piece of infrastructure suddenly failing Microsoft Health checks for proxying requests to the backend. While this was resolved quickly, diagnosing it took a while. All users had access to ExchangeDefender LiveArchive during the work and all services are back to normal.

07/27/2020 13:58 PM 4 days

The issue with OWA/EWS has been addressed, and all users that were on the impacted are now reporting connectivity and routing is back to normal. 

07/27/2020 13:38 PM 4 days

Some users reported they are having issues connecting to OWA and Outlook. We are investigating what could be causing the issue, please stand by.

We'll update this SA as soon as we have more information.

Thank you for your patience.

Outlook crashes as soon as it's opened (Resolved) Investigating / Notice
4 days

Microsoft's latest update to Microsoft Outlook is causing issues on some workstations. Essentially, Outlook starts up and crashes immediately, and running it in safe mode or in an isolated/compatibility mode doesn't fix it (as it did in the past when this issue happened).

The only thing you can do if you experience this issue is to roll back the Microsoft update to the previous release. To do so, run a version of this command with paths appropriate to the way your workstation is configured.

"C:\Program Files\Common Files\microsoft shared\ClickToRun\OfficeC2RClient.exe" /update user updatetoversion=16.0.12827.20470

For more information, and further troubleshooting in the wild, take a look at the following reddit thread:

https://www.reddit.com/r/sysadmin/comments/hrq0mn/outlook_immediately_crashing_on_open_after/

This is a software issue with Microsoft and is not something under ExchangeDefender's control or ability to assist/fix, until Microsoft gets this repaired we are recommending either rolling Outlook back a version or using our LiveArchive or Outlook Web Access from your browser or mobile device.

DNS resolution failure on outbound network (Resolved) Issue / Performance
6 days

We are getting reports of rejections on the outbound SMTP. The rejections are due to a DNS failure (reverse DNS lookup / PTR) on one of the ranges we use to proxy deliver email. As of 6:11 AM this issue is resolved and mail is moving without error. If you got the NDR, please resend the message and it willl go through normal delivery process bypassing the affected range.

Exchange Upgrade Finalizing (Resolved) Outage
30 days

We have resolved all the issues and completed all migrations and cleanup.  Everyone is now on the new tech and everything is working correctly.

If you moved away from ExchangeDefender and still need some data, it will be live for another 60 days on our legacy platform, here are retrieval instructions

If you would like our assistance, please let us or your IT provider know.

06/19/2020 20:43 PM 11 days

We wanted to offer one final update before we close the ExchangeDefender NOC covering our Exchange migration.

The past few days have been largely consumed with cleanup and misc configuration requests already covered here. By far the biggest issue has been reseeding and legacy copies of mailboxes exceeding 25GB using nearly all internal, Microsoft/powershell, and third party tools there seems to be no predictable, foolproof, failsafe way to migrate a mailbox. The larger mailbox gets, the more difficult it seems to port (one particular user has been waiting on their mail for 2 weeks - they have a 70 GB mailbox and it's taken dozens of attempts of repair/recheck/export/move/seed/verify) and it has been the greatest source of frustration for us and for our clients, largely because the progress indicators are unreliable and process very prone to failure the larger the mailbox gets. This is why when we started offering 2016 years ago we set up the 50GB quota with 15GB realtime and 35GB in place archive setup so we can deliver on both service restoration and disaster recovery.

We are continuing to assist our partners in the following areas:

  • Outlook connectivity (if it keeps on prompting you for a password you need autodiscoverregistryhacks.zip)
  • Distribution Group (External) and External Forwards UI (we discovered a bug, the control panel will be back over the weekend and in meantime we'll create it for you manually with a ticket request)
  • Cancelled services (as of yesterday 6/18 we have the ability to remove organizations from ExchangeDefender/O365, so if you client cancelled or went to another service even within O365, open a ticket and request that we delete the org. You can do so on your own as well if you've deleted
    all the mailboxes/forwards/groups.)
  • IoT/SMTP (while Exchange/O365 does support SMTP connectivity, managing it through our IoT connector is far more secure and
    reliable)
  • Implementation of Shared Mailboxes. Please, please, please, please DO NOT use Public Folders anymore, for any purpose. Create a Shared Mailbox instead.

At this point everyone can connect, mail delivery and legacy reseeding are in progress, all systems for Exchange, ExchangeDefender, and
LiveArchive are working normally.

We're looking forward to closing this ugly chapter. We have done everything in our power, and we couldn't be more thankful for our partners who have helped us with the cleanup of the Microsoft disaster. Thank you. We are sorry that so many clients were inconvenienced with this, we planned and managed every step of this migration by the book with thousands of other successful migrations that happened from 2016 Aug 2019, but when your vendor pulls the rug underneath you and damages hundreds of mailboxes unannounced many of us will soon be enjoying the first day off in June. The only good news is, you will not have to go through this process again. 

06/16/2020 19:18 PM 14 days

We wanted to offer a major update on the migration, specifically covering the major issues we have addressed for some clients during the cleanup phase. 


Distribution groups, forwarding - We have received reports from several organizations regarding issues covering distribution groups, group members, forwarding account directions (forward vs. store & forward). If any objects failed to import due to configuration/contents/policy/etc it is in the retry queue and will be published shortly.


Add / Delete Mailbox - We have addressed a bug in the add/remove process that was prohibiting certain organizations to add/remove accounts. Originally, as noted on anythingdown.com NOC, we blocked this function entirely because users were looking at an empty list and creating mailboxes (that would cause a collision when the new mailbox was migrated from the source). This problem is fixed, if you encounter an issue please open a ticket with a screenshot and as much info as you can provide.


Add / Delete Organization/Domain - At this moment it is not possible to add/remove organizations, or those that were in the system recently. In order to finalize the migration, the routing policies are locked down (meaning if you deleted a domain, ExchangeDefender will still treat them as local). We look forward to wrapping this up shortly.


Password / Login issues - This is by far the biggest ticket group category, we are still processing double digit requests for credentials, credential resets, and credential tests. Similar to the next group:


Outlook issues - We are still spending a lot of time going through the basic Outlook configuration steps. For an overwhelming majority, this transition has been transparent. Those that did not and had to take a manual configuration route, the process has been described at anythingdown.com 1) Make sure you have an autodiscover record 2) Make sure it propagates, then run the autodiscoverregistryhacks.zip 3) If you don't control your DNS, make adjustments to your local systems hosts file 4) Setup Outlook with autodiscover, the UPN must be used as your login address if you've changed it from your primary SMTP address.


Missing & Syncing Emails - Every mailbox that has been reconnected has either had all it's mail delivered directly, delivered in a Catchall account - user@domain.com. Some users are confusing items they see in their Inbox in LiveArchive but not in their Outlook/OWA (but after extensive searching we keep on finding missing messages in folders, Deleted Items, etc). If something is missing and absolutely critical in LiveArchive just click on the message and click Forward to your email address and the message will be forwarded to your Outlook/OWA.


Store & Forward - Several users were also unfortunately caught up in a custom policy that did not get migrated to the new Exchange. These are more legacy configs we did for some users in AUDC, things like renaming the OU or primary domains. For some of those accounts, the store and forward rule because a forwarding only rule, skipping the Inbox and going straight to the person that it's being copied to. We have fixed this issue and it should not be happening again.


Autodiscover - We have gotten several complaints about autodiscover. Microsoft has removed manual configurations in 2013 and no modern version of Microsoft Exchange supports a manual server setup. However, this is something that could be easily rectified even with minimal technical skills by modifying the local hosts file if you don't have the credentials to do it properly by modifying the DNS. Absolutely everything in the new infrastructure relies on the autodiscover record!


iPhone / Android Setup - For the most part, we are just confirming that all mobile devices should work fine with owa.xd.email as the server name, ditto for EWS integrated applications, we have not received more than an inquiry for the server name. For Android, things get sketchier when you consider all the different vendors, apps, and configurations. Again, so long as autodiscover is present and configured properly and your device is using a modern client, it should just work. When it doesn't, recreating it takes a few minutes.


NDR - Non delivery receipts and errors are always of high interest to our NOC team as we continue to go through cleanup and audit all the tickets and users.


These are the issues we are currently working on, in 3 shifts, and sorting them all out as fast as possible. I know that for many of our clients this transition has been messy, but you are on such a better and more secure platform that won't require you go through this process again. While modern platforms are more secure, their recovery from a disaster or issue (as some of you unfortunately went through) is extensive and at times unpredictably slow - so you have this much of a committment for us, we will make sure LiveArchive is able to step in on a whole new level when things like this happen.

06/16/2020 07:53 AM 15 days

We're in the final stretches of importing public folder data from the SCROOGE, LOUIE and GLADSTONE cluster. Folders from DARKWING  were successfully imported. Public folders from ROCKERDUCK will be exported in the next batch along with public folders with more than 5000 items.

06/15/2020 10:11 AM 16 days

We are near finished with completing the first batch of data imports for public folders. We will be remapping identities and public folder permissions once the import completes (estimated around 730-8AM Eastern). We are targeting making the first batch available by 10AM Eastern

06/15/2020 06:04 AM 16 days

Public Folder Update

We're in the progress of migrating public folder data. We fully intend on restoring all public folders with less than 2000 items by the start of business on Monday June 15th. However, we are not imposing the item limit to aptly named company calendar and contacts folders. Unfortunately there are a lot of public folders that were previously renamed in Outlook (on the legacy clusters) which is making remapping of the new identities difficult. 

Public folders with more than 2000 items will be imported after the initial batch import completes. 

Public folders will automatically become visible as additional mailboxes in Outlook and can be mapped in OWA by adding the public folder to the favorites. Public folders will be renamed by prefixing the org to the public folder name, removing the MSP name and the customer domain. For example, the previous public folder "demodomain.com_public" for the org demo will become "demo_public". We understand renaming the public folders may require some minor changes in LOB software, however, the renaming is required to ensure uniqueness of the public folder name.

Partners should expect to see public folder permission management inside of service manager by the end of the week. 

We are taking requests to prioritize public folder imports. If your customer does not see their public folder by the start of business, please open a support request titled "Public Folder: expedite" . Inside the service request we will require the domain name of the customer requiring public folder data. If you happen to know the name of the specific public folders to expedite, they will be accommodated in the request. Customers with more than 2000 items requesting expedited access to public folders will have their expedited request filtered to mail within the last 90 days and the rest of their data will be imported in a subsequent batch.

06/10/2020 15:45 PM 21 days

11 AM Update

Mailbox Data Reseeding

At 10PM last night we picked up the pace of email delivery for users whose automated Exchange migration failed. If you had a user in that situation for days, we have created an empty mailbox for them so they can start working, and their old mailbox is being imported in the background. Within a day, it will look exactly the same as the old thing except much faster, safer, more secure, and on the new generation of Exchange. 

Account Connections

We are still assisting clients with password resets, account reconnections, misc Outlook issues and mobile. At this time there is nothing else to report on this front as all the issues are addressed following the instructions already mentioned in this NOC advisory. Outlook, mobile, OWA, EWS, printer/CRM connections, everything works. This is by no means a new system for us, remember that we started offering Exchange 2016 in 2016 and have been facilitating migrations to it ever since. While the disaster has been in the migration failure for so many, the experience on the new platform is not raising any issues. We're even slightly hopeful because many of the reported issues clients had in Exchange legacy (connectivity, timeouts, email delays, etc) have not been a problem on the next generation so far.

Next Steps

We will likely spend bulk of Wednesday and early Thursday completing mailbox reseeding, redelivery of mail from 6/1 - 6/9, and getting everyone connected. Thankfully LiveArchive has been a life saver and people have been able to work from there, we want to make sure everyone is back in their Outlook, OWA, mobile experience and we can move forward.

Public Folder data sync and reconnection will happen closer to the weekend, at this time we do not have a firm time frame but the process has been scripted, tested, and certified for months so we do not expect issues there.

We will keep you up to date here but PLEASE if you still are not able to get into your Outlook Web Access at https://owa.xd.email/owa open a ticket with our team and we'll help you get reconnected and going.

06/10/2020 03:07 AM 21 days

11 PM Update

We are starting to replay more mailboxes faster and get everything in sync. 

At this point the next stage of migration recovery is largely automated and one-on-one for weird Outlook/Sync problems. We will not have further updates from the migration team on this ticket until 11AM.

06/09/2020 18:28 PM 21 days

We are still working on password resets, last week mail sync, and old mailbox sync for those that failed Microsoft's automated migration. 

If you are at this point not able to get into Outlook Web Access, please open a ticket and list which email addresses that are not able to login to OWA (one per org)

06/09/2020 07:40 AM 22 days

3:30AM 

We are still wrapping up tickets from the day, a rather busy Monday getting everyone sorted out with Outlook cleanup and resync as required.

If you encounter any problems and are updating tickets, PLEASE provide screenshots and/or detailed error/log reports if you possibly can. Our support front line typically collects the information, double checks it, passes it on back to NOC or network admin on duty for resolution. If the error is vague the wait time and delay can depend on how long it takes support to actually determine the issue. So for example, if you're having login problems please specify your credentials and the site along with the error that you get. If you are getting mail delivery errors, post the NDR (non delivery receipt) or anything that will help us get your account and issue sorted out faster. 

06/08/2020 23:36 PM 22 days

7:35 PM 

We are still replaying last weeks email into all affected mailboxes. You will see a "CatchAll-Import user@domain" folder with messages that were delivered before your mailbox could successfully go online. 

06/08/2020 23:13 PM 22 days

Monday

Overall, things are going in the right direction and we're going through typical Outlook cleanup, some reseeding, etc.

We have spent the day resetting passwords and helping clients reconnect their Microsoft Outlook, etc. Here is what we're seeing and hearing:

1. Many accounts need password resets. About half the work we've done today in support has been a simple password reset, so if your account is not letting you in with credentials you know, please reset it and give it 5 minutes to sync up. If that doesn't work, open a ticket immediately with the email address as the subject. If your mailbox is stuck in the password reset request, likewise, open a ticket.

2. Questions about Exchange autodiscover. No, you do not need to change your autodiscover if you've already got one. Yes, you absolutely need to have an autodiscover record for 2016 and above there is no "server name" because Outlook will not resolve it for configuration (it's outlook.xd.email). For mobile devices and anything relying on Exchange Web Services you can use owa.xd.email. 

2a. Problems with Outlook. Please read https://www.exchangedefender.com/blog/2020/06/exchangedefender-exchange-setup/ almost all Outlook tickets are addressed with this. If you can login with OWA, if you have run autodiscoverregistryhacks.zip batch file as Admin, if you have Outlook 2010 with latest patches or newer - you should have 0 problems connecting to OWA.

3. Empty mailboxes, missing last week, etc. There is no reason to worry, we have mailboxes archived 3 different ways so if you can get into LiveArchive or had a domain org, we've got your email. We're replaying it but it is moving slooooooow. Essentially the process searches a temporary mailbox, locates messages sent to a specific primary email address, and then pushes those messages up. For blank mailboxes, described yesterday, they will be reseeded as well. Everything is moving in the right direction, we just need time. We're pushing the system to the limit in terms of traffic.

4. OWA "Lite" mode. Please do not use popup blockers on Outlook Web Access. If you do, it will block several critical components OWA has and it will launch in a light mode that looks like OWA from Exchange 2007. Just add it to the trusted sites or disable popup blocking on it. The theme can be managed through Outlook settings.

5. We have seen some NDRs that we are still investigating. So far we can explain away all of them (DKIM errors from LiveArchive, missing aliases, old organizations previously removed from Exchange org recreated because the migration has been staging since November!)

6. We have seen a few mobile device reports (calendar sync) that we are still investigating. We as of yet do not know if this is an issue, if the Outlook app is reprovisioned/reconnected properly, etc. In general, all mobile devices should use the owa.xd.email as the "server" and their UPN (login) and password.

06/08/2020 00:47 AM 23 days

9 PM Update EST:


At this point we have moved, created, troubleshooted, and sync'd every clean mailbox in our enterprise and all accounts are active and working. 

Any mailbox that we could not successfully autoreconnect to the new organization is being created from 9PM to 11PM and mail will be replicated into it manually. 

This move assures that access to email is restored for everyone on our platform, regardless of Outlook/Exchange instrumentation problems. It also puts every inbox into the service manager and gives our clients the ability to manage their mailbox immediately. If you're having issues, create a new profile, reconnect the mailbox and all your data will sync down either immediately for active accounts and shortly for accounts that have been reported as inaccessible via OWA. We are doing this so that everyone can use OWA and Outlook immediately.

If you choose to setup a new profile, you're set. If you do not create a new profile, your Outlook will start up and give you two options "Use Temporary Mailbox" or "Use Old Data" - please click on "Use Temporary Mailbox". Here is a pic of the screen for reference:

https://www.exchangedefender.com/OUseTempMbox.png

06/07/2020 21:27 PM 23 days

5:30 PM EST:

We are continuing to go through accounts of everyone that reported an issue. IF you reported an issue OR IF we are unable to login with any legacy/2016 credentials, we are continuing to work through those accounts and your mailboxes WILL be online tonight.

In terms of setup, the servers are owa.xd.email and outlook.xd.email, but neither is neccessary if your autodiscover is set. If you have/had an autodiscover, it does not need to be modified, all the autodiscover instances point to the new infrastructure. Make sure you've run autodiscoverregistryhacks.zip as the Administrator on the workstation, reboot and Outlook will restart. If Outlook was open the whole time during the move, you may get the prompt saying "The Microsoft Exchange administrator has made a change that requires you to quit and restart Outlook." but the result will be the same, Outlook will restart on the new infrastructure with your folders and new email in the Inbox.

06/07/2020 17:05 PM 24 days

Noon:

We are continuing to go through all the tickets and confirm / double check every account. We've been following up with clients that have contacted us via tickets, Facebook, SMS, email and providing passwords, reconciling the view in the Service Manager. 

We are anticipating that this process of double-checking every single account will be completed by 3-5PM EST today at which point we would have confirmed everyone that has raised an issue is online (and many others that have not). 

Mail flow to everyone is resumed as soon as their mailbox is online and redeliveries are happening so we're confident that as soon as we clear the list above we can say everyone is online and running without issues.

06/06/2020 14:34 PM 25 days

10AM EST

We are continuing to work on the tickets and reports of users that have not been able to login, many of these have been resolved already by our partners (by changing the password) but we will not leave a single ticket alone until we have been able to confirm everything is operating normally. This is an extremely important step in a migration because small issues in migration can cause larger issues down the road. (many of the accounts that did have problems had them because of a very custom / complicated configuration such as having multiple domains in the same organizations randomly used for authentication, clients opening  5 mailboxes in a same Outlook profile and typing in the wrong password, odd permissions, etc.) This is not to place blame on anyone that had such a configuration, we understand that every business has legitimate business process and operations needs, but when Microsoft & support tools we have at our disposal decide to stop supporting such configurations we have no easy means to recover them and it becomes a hours long ordeal or surgery.

We are still going through the tickets at this hour, we are waiting for the last few mailboxes to complete retry/reseed/retransfer/redo before we apply recipient policies so we can redeliver email that went into catchall mailboxes, intermedia failover authorized relay queues (so we don't bounce stuff that can be delivered), etc. None of these "missing" items are missing from LiveArchive, so if you're missing anything keep in mind you've had a LiveArchive mailbox since the day you signed up for ExchangeDefender for occasions exactly like this one. We have even deployed free LiveArchive for clients that purchased our barebones entry level Essentials products.

We are continuing to work on this and will provide updates as well as the timeline because the next steps are not reliant on using Microsoft tools (no PowerShell scripts with no progress or status indicators). 

We have been heistant to introduce any other workarounds during this window as we were completely blindsided and needed to recover full access to as many people as possible. This will never happen again. Over the weekend we are working on some solutions that will greately expediate the self management of these mailboxes and authentication/ID processes. 

06/06/2020 06:55 AM 25 days

Midnight

Most of us are still here going through tickets and double-checking every failed login, mail bounce, and recipient policy. This part is currently underway and we anticipate completing it by 8 AM EST when the next post will be published. This step is critical in applying all the missing distribution group, alias, security policies, public folder permissions, folder mapping, etc. At that point, all "missing" mail that was not delivered to those aliases/dgs will be replayed and delivered.

We are still monitoring a few mailboxes that failed import/connection to the appropriate Exchange organization. Exchange mailbox import can fail after a certain threshold of bad items in the mailbox (we started with 50 and are currently at 100). Because all of these mailboxes currently have LiveArchive and can work, we're resyncing some of them.

We continue to see tickets regarding Outlook connectivity. If it works in OWA, it will work in Outlook. If it doesn't connect in Outlook 1) Download autodiscoverregistryhacks.zip, run "cmd" as Administrator,  reboot 2) Confirm the workstation can resolve autodiscover.YOURDOMAIN.COM 3) Check OWA at https://owa.xd.email and confirm that you have the right credentials 4) Open your Outlook (2010 with all service packs or newer) 5) Start Outlook. You will be prompted to accept new configuration, or "repair/rebuild mailbox", or "follow mailbox to the new server" and after accepting all of that... your Outlook will open up, sync up all your folders and you're back where we started just with the most up to date version of Exchange possible.

The next update will follow at 8 AM EST. We are continuing to work through the weekend and hope to be able to sleep soon. Thank you for your patience and kind words through this process, we realize this is a giant inconvenience and a business interruption.

06/05/2020 22:37 PM 25 days

Update 6:30 PM EST:

We have spent the entire day troubleshooting mailboxes across domains. The most common issue is the login mismatch, where users are using the wrong UPN or wrong username. We have been troubleshooting some tickets with missing aliases / distribution groups (those will be reapplied and mail redelivered), etc

We have been updating clients throughtout the day so if you opened a ticket this week you will get an update on your user/domain as soon as we confirm that they can access Outlook / OWA (this has been our protocol throughout the week).

Next update will be tonight at midnight, we are currently working on mail routing reports, troubleshooting reported failed OWA/Outlook logins, PF/DG/AL outstanding imports/issues/debugs.

Thank you for your patience. We are moving through this slow and tedious detailed work as fast as possible, every issue in this migration will be addressed. If anyone has ever unplugged a cable during a migration, PC update, or similar that's the disaster we had to clean up (but with certificates, authentication, routing, etc) that was crash dumped into our lap.

06/05/2020 17:07 PM 26 days

Update as of 1PM:

The only issues we are still seeing is related to authentication, password failures and resets. Team is moving through them quickly. 

Several Outlook tickets came up, please follow directions in this post (https://www.exchangedefender.com/blog/2020/06/exchangedefender-exchange-setup/). If they do not have Outlook 2010 or newer, if they do not run the autodiscoverregistryhacks.zip file on their PC with administrative privileges, if they do not reboot, Outlook will not connect to Exchange.

06/05/2020 12:01 PM 26 days

7AM EST

As of 6:23am all mailboxes have been reconnected in the new environment. Everyone should be able to access their mail via Microsoft Outlook, Outlook Web Access, LiveArchive, etc. The instructions on how to handle misc issues is in this NOC post. 

As of 5:30am all mail routing issues have been addressed and ExchangeDefender is delivering to all platforms as usual.

All access is fully restored to Microsoft Outlook, Outlook Web Access, and recovery systems.

We will be spending most of the day working wth clients and partners with login issues. Most people do not know their Outlook password, and misc issues surrounding Azure AD and AD congestion have slowed down credential resets earlier in the week. Today our final priority is helping the users with these authentication issues.

06/04/2020 22:28 PM 26 days

6PM Update

We are continuing to go through misc Exchange issues and mailbox reconnections. We believe we have at this point sorted out all but the following service items which we are working on at the moment. The biggest issue today was continued latency in Active Directory, something that we've addressed and solved as of 4:30 PM. Here is a summary of what we have left:

Mail routing and delivery

At 6PM we solved the issue of roundabout mail routing for mailboxes that have not been connected yet. Because the rug was pulled under us on Sunday we have been exceptionally aggressive towards archiving - so any mail that couldn't be delivered to the primary email address would be routed to LiveArchive, to catch-all domain mailbox, and a failover node. 

Items in LiveArchive that are not in OWA/Outlook

This is related to the routing/delivery of the mail between various failover systems. Good news is because we have written LiveArchive for this very reason, we will be able to sync Sent Items right into their Exchange mailbox.

Mailbox moves and remounts

We are doing the final sweep of mailboxes that have failed to mount. We still have a few partners that are affected. 

06/04/2020 13:55 PM 27 days

Important: 

0. Download and run the registry files as the administrator, regardless of whether you already have autodiscover or not. The file is here:  https://www.exchangedefender.com/media/autodiscoverregistryhacks.zip 

1. If you already have an autodiscover record DO NOT CHANGE IT. We are

2. If you do not already have an autodiscover record, YOU DO NEED TO CREATE ONE. autodiscover cname autodiscover.xd.email. 

The trailing dot is important, depending on your DNS software. You can tell if you've done it correcltly if your DNS lookup looks somewhat like this (from Windows 10 command prompt):

nslookup autodiscover.ownwebnow.com

Server:         65.99.255.161

Address:        65.99.255.161#53


autodiscover.ownwebnow.com      canonical name = autodiscover.xd.email.

autodiscover.xd.email   canonical name = autodiscover.dal.xdmail.online.

Name:   autodiscover.dal.xdmail.online

Address: 72.249.54.208


If the autodiscover is missing, you will get an error instead:

nslookup autodiscover.xdref.com

Server:         8.8.8.8

Address:        8.8.8.8#53


** server can't find autodiscover.xdref.com: NXDOMAIN

----

To figure out if your workstation can detect the autodiscover record, Start > cmd > nslookup autodiscover.YOURDOMAIN.COM (substituting your domain name for it)

06/04/2020 13:30 PM 27 days

9AM Update:

We are moving through our tickets, resolving any outstanding issues, and keeping people in the loop here and support.ownwebnow.com

1. One of the things we are noticing more and more is that some people do not have autodiscover setup. If you can login to Outlook Web App (https://owa.xd.email/owa/) or NGE, then your credentials are good and your mailbox is online; so if Outlook continuously prompts for a password or takes forever to start up a session, confirm that you have an autodiscover record and that you have run the registry hacks. The process is described here: https://www.exchangedefender.com/media/ExchangeDefenderExchange.docx

2. Overnight we added some more horsepower to our LiveArchive NGE, it's actually moving faster than Exchange at this point so we hope the improvements make it easier to work. 

3. We were able to identify and fix the internal routing issues, some clients reported that they saw some messages in LiveArchive that did not show up on the Exchange side. The same issue affected many login failures (from time to time, not completely), some mail delivery, (it's in the catchall), etc. We believe the changes that were made overnight will allow user logins to proceed without an issue.

4. If you have an Office365 version of Outlook, or if you setup your outlook 5-10 years ago with autodiscover but are now running on the latest one, you will still need to get and run as administrator (and reboot) in order to enable non-Microsoft Exchange servers. If you HAVE autodiscover and you are still getting repeated password problems, please follow the instructions in our guide on how to download and execute the patch, reboot, and after a minute or two it should be fine.

5. If you've never had Autodiscover, manual server configuration has been decomissioned by Microsoft in Exchange since 2013. We recommend setting up a new mail profile (Start > Control Panel > (Click View > Small Icons) > Mail (Outlook 20xx) > Show Profiles > Add) and you'll be able to setup a new autodiscover profile without destroying your existing one. 

6. We are preparing to roll out swing mailboxes for the few mailboxes that we still have not been able to attach. 

7. Staff will be spending most of the day following up. If you can, please keep all ticket updates on the same ticket, we are going through issues domain by domain.

8. Please use OWA if possible, please use NGE as possible.

Next update will follow later today but 

06/04/2020 07:06 AM 27 days

3AM Update:

We are continuing to work with our partners and clients on getting users to the modern Exchange standards. Microsoft no longer supports "manual" server configurations as of Outlook/Exchange 2013 so if you do not have an autodiscover record you need to create one immediately (simply create an autodiscover CNAME record in your domain with the value "autodiscover.xd.email"). We know that many of our partners and clients aren't highly skilled in obscure Microsoft Exchange settings and configurations so we've written up a quick guide that just about anyone with Admin access to their PC or mobile device can follow:

https://www.exchangedefender.com/media/ExchangeDefenderExchange.docx

Please feel free to distribute the document or customize/brand it to your liking. The process is very simple, as long as you can login to Outlook Web App (https://owa.xd.email/owa/) those same credentials will get you into the new version of Exchange. 

If you have autodiscover configured already (you should, without it even 2013 and earlier barely worked consistently) your Outlook will prompt you for your password. You may get an error or warning and within a few minutes a screen prompting you to accept new settings from "outlook.xd.email" will show up. Click on accept, Outlook will open your mailbox and everything is done. If you experience a problem in this stage, start Outlook in debug mode (outlook /rpcdiag) and observe the servers it connects to - if it's Office365 you will need to follow the same steps in the guide for running Outlook Registry Tools. 

In terms of backend work, we are still helping partners mainly with authentication and mail flow problems. We still have a few users that are failing auth because of all we've discussed before, we're confident these last few changes will make things move smoother. 

The most difficult issue for us remains communication and ability to update so many clients and partners, and troubleshoot smaller issues with individual mailboxes and organizations. We remain confident that even though we've been dealt this setback, we will have everyone on the new platform and it is well worth it. Next update will be provided at 9am.

06/03/2020 20:02 PM 27 days

4PM Update

We are continuing to see the NGE environment normalize and people return to work. There are still some issues that we're trying to get to the bottom of (already covered in previous NOC posts, Outlook/autodiscover/password resets just not being as consistent; we have been troubleshooting these on the backend). For the most part, everything is working - and we have a number of users that are still not online that we're going above and beyond to restore the service where Microsoft tech is simply incapable of doing so (see below under IMPORTANT - PLEASE READ if you're still down)

The biggest issue at the moment is on the staffing and technology side. None of us have left since Sunday and our original migration and transition plan had to be compressed down and executed quickly. Our support is simply not capable of handling tens of thousands of tickets so we have been consolidating them per MSP and per domain and trying to get as many up and running as fast possible (see TECH DETAILS section) but some clients are understandably frustrated with the outage and creating dozens of tickets which is only slowing us down. Requests for a callback, to recite what we have in these NOC updates, is also slowing us down. We are on your team, we have your best interest in heart and it does not help us at all if your clients leave you. We take our responsiblity to our clients seriously and are doing everything we can to get everyone up and running - but we're overwhelmed and exausted and beaten and I'm sure everyone in IT can relate to that. We're working on it. Below is a brief technical summary for the outstanding clients that are down, an explanation, and a workaround.

TECHNICAL DETAILS - IF YOU ARE STILL NOT ABLE TO CONNECT

The remaining mailboxes that are still not online fall into two technical categories. 

1) They have moved/moving and the O365 side has failed to reconnect the mailbox to the domain or we are attempting to move a mailbox and it errors out and fails after it reaches a failed item treshhold. The reconnection process and seeding process fails and we go through a manual attempt through PowerShell to map the mailbox to it's proper location so Outlook/OWA/EWS(mobile) can connect to it. There are tons of different technical issues here (from not meeting basic password complexity, one user had P@ssw0rd as a password - to bad X400 address / primary address mismatch / etc). Rinse, then repeat, over 700 times since Sunday. We're down to only a couple that are in this stage.

2) They are still in process of moving - These are mailboxes that were never flagged for move because they were renamed, had weird / circular permissions or other misconfiguration, OR, they got disconnected from the move when the source was disconnected. For these, we are moving them to production but some users have 30-40GB mailboxes and the process of migrating is slow. Once they are moved, we go back to step #1 and confirm that they can login. Then someone in support has to dig up the correct ticket and update it. 

We are currently working with partners over the phone, over our NOC, over Facebook video/chat, over txt, and we're doing everything in our power to connect the remaining accounts. This is brutal and difficult and we understand how bad it is - and we are here working for you and trying everything we can to get everyone online as fast as possible. This whole episode was not our plan but Microsoft pulled the rug from under us and we're doing insane stuff just to get everyone working. See below:


IMPORTANT - PLEASE READ - IF YOU'RE STILL DOWN AND WANT AN IMMEDIATE WORKAROUND

If you do not want to rely on LiveArchive or NGE, or wait for the mailbox to be repaired/restored/reconnected/etc we can bring you to operations quickly - but fair disclosure, this is ugly yet functional. The following process can be executed by our NOC to create a temporary mailbox that we can then merge together when the original mailbox is ready.

Create a new mailbox with a unique email address swing.USERNAME@domain.com. On the edge transport servers we will create a rewrite rule which says mail from the swing address gets rewrote to the real address and mail to the real address is rewrote to the swing address.

The user will have the same display name with an extra space so Vlad Mazek => Vlad  Mazek

We create a transport rule for intra company mail to redirect to the swing user.

At this point, all their mail flow is working fine and they can work out of OWA.

After completion

Once the migration is completed, they’ll be closed out of their  swing mailbox (or we can give it full access permission to the real mailbox so it automagically shows up in owa/outlook). The real account is then active for login and we disable the rewrite rules and import the data from the swing mailbox to the real mailbox.

06/03/2020 14:31 PM 28 days

With almost all mailboxes reconnected, we are now addressing mailboxes that are in a failed/disabled state. These mailboxes are disabled for a variety of reasons (account or domain mismatch, account deleted, not in the right security group, etc) and we're enabling them, clearing any errors, reconnecting.

We already know which mailboxes are in this state and we're working through the list, in the meantime please stay in LiveArchive if your mailbox is affected by this and we will update the existing tickets as we go along.

I know many of our partners are asking for an update and specifics to pass to their clients and we'll offer a detailed writeup by noon on www.exchangedefender.com/blog


06/03/2020 08:33 AM 28 days

4:30 AM EST - The mailbox process is nearly complete, we have resolved a number of reported issues overnight as well. Our next step is to deliver PF and catchall mailbox access.

06/02/2020 21:38 PM 28 days

We are at the tail end of getting everyone off the legacy platform, and we have restored access and mail flow to just about everyone. The last few mailboxes are exceptionally difficult and we are doing everything in our power to get them completed before midnight, but a few may slip. Here is a summary of work and issues we've resolved throughout the day in case you expereinced any of these problems we hope you can check again:

1. LiveArchive NGE DKIM signature - our infrastructure was not signing DKIM messages, in interest of expediency we've routed outbound mail via ExchangeDefender instead of implementing DKIM on the platform for all of our client domains.

2. Occasional 504 gateway timeouts - these typically happen when our backend docker services do not respond in time to the frontend proxy. We've bumped the resources and timeout settings there throughout the day and it's performing well considering exceptionally high usage.

3. "Report Issues" button in Service Manager - several of you have found it, it's something we've been trying to sort out throught the day so we can avoid doing individual service requests through support tickets. We can automate a lot of the discovery and analysis through this and get users problems autofixed going forward. Please note that this is not going to help mailboxes that aren't already mounted, this is more for the ongoing support.

4. Active Directory resiliance - we are adding more domain controllers to the mix. These will not go online till midnight for our clusters located in USA, and until tomorrow for EMEA/AUS.

5. We are sorry about this, we understand the frustration and we're doing all we can as fast as we can to get everyone rolling in the right direction. The rug was simply pulled out from underneath us with no warning and our actual migration checkup plan that was supposed to last 2 weeks on 6/15 had to be executed in 1 day.

Next update will follow at 9:30 PM EST.

06/02/2020 16:37 PM 29 days

Noon update: We are in the final stages of reconnecting and disconnecting all the affected mailboxes (some larger mailboxes are requiring a reseed so if you haven't gotten your users and mail enabled public folders onto LiveArchive, this update is specifically for you). 

https://www.exchangedefender.com/blog/2020/06/update-exchange-migration/

We have also addressed an issue where catchall mailboxes (catching/caching mail for inbound) are accepting messages but still issuing an NDR. Followup post will explain how to access that mail shortly. 

We are optimistic about the current progress and believe we've addressed all the unexpected issues that have come up since Sunday. We will keep you up to date on our progress on the NOC site at www.anythingdown.com

06/02/2020 13:04 PM 29 days

9AM 

We have largely restored access to practically every mailbox that had a simple fix and are now making additional passes over mailboxes and organizations that had custom configurations or other errors preventing them from reconnecting manually. The process of updating all tickets, clients, and mailbox configs is manual and time consuming so we've been working on a few things that will be announced ~10AM EST today. 

1. We are prioritizing getting LiveArchive NGE (launched last month) access for users that are still affected. We will have a quick walkthrough on how to create accounts for public folders/etc so at least people that are still affected can get email.

2. The manual reconcile process is moving forward.

3. No additional issues have been reported, which is the only good news to report since Sunday, and our goal is to make sure EVERYONE can quickly recover 

4. All tickets are being consolidated per company so we can keep you up to date better and we're discussing our update policy. We're all in the same boat here, while we are working as fast as possible and realize that everyone has a critical issue, everyone is upset or livid (as are we) --but we can either work on getting you online or we can deal with "ETA", "What do I tell my client", "This is still not working" ticket updates that are just putting all of us at a disadvantage. Priority is getting people email, there will be plenty of time to make everything right and reconcile things.

5. New UI and LiveArchive tweaks are being put in to allow people to manually address LiveArchive login/etc separate from Microsoft Exchange, so that mailboxes that are still affected and so that users that live in Public Folders or have mail flow interrupted for whatever reason, can continue to work.

We thank you for your patience and are working as hard and as fast as possible. 

06/02/2020 04:11 AM 29 days

Midnight update:

We are still going through cases and repairing/activating in the debug queue. More accounts are reconnected and verified, we are working through the queue. If you've reported the issue, or if we identified an issue with an account, we are working on it (and tickets will not be updated until either there is a new NOC update with our progress or we have verified we can login and manage all accounts in the domain).

We hope to have all issues (except Public Folders) addressed shortly, we will have a new process for reporting outstanding issues in the AM for users that are still reporting problems.

06/02/2020 00:10 AM 29 days

Update 8PM EST:

We are currently working through a project queue that our partners and support team setup throughout the day to address any outstanding accounts that are having login issues. We are prioritizing authentication requests at this time and will be working through the night as long as it takes to get everyone into their Inbox. At the moment, the second biggest issue we are working on is related to reattaching Public Folder infrastructure to some organizations. We will be updating support requests as we reconnect entire organizations, and will kickstart PF reconnection afterwards.

More details: Remaining mailboxes that are currently inaccessible and password reset does not resolve the issue did not successfully move to the new organization and disconnect from the source (problem causes involve Active Directory errors, Exchange errors, password complexity) and the recovery process is to simply check the error, make required adjustments, and attempt to reconnect the mailbox. Because each issue is separate and caused by different factors, this is a slow manual process. We are working on it and anticipate everyone will be reconnected/restored tonight. There are several mailboxes in the move that were never picked up by the Migration Engine and we're manually moving those accounts too but some of the larger mailboxes are taking time, thankfully there are rather few of them and we have tons of resouces.

We will keep you up to date throughout the night and we are very sorry about the inconvenience this has caused our users. We're doing our best to get everyone up to speed and we realize that everyone has an urgent need to get back to their Outlook, we are here for you. In the meantime, we hope you can continue to work out of LiveArchive at https://nge.exchangedefender.com as we get this up and running smoothly, the level of complexity and issues with the legacy Exchange platforms has been significant and we have planned this process for months and executed countless test runs with very few issues. We realize that it doesn't matter how thought through and designed the process is if you can't login to your mailbox right now and we're continuing to work on this. 

06/01/2020 19:43 PM 29 days

Update as of 3:30 PM EST:

All but ~200 mailboxes have been restored, mail flow has been showing as nominal since it was activated this morning. The mailboxes still in manual debug are on organizations that have changed UPN or had some more complex settings that we are reconciling and activating manually. 

1. The biggest update since the last NOC is that all organizations have been moved to our next gen Service Manager. If you see your organization in there you should be all set and all functionality (password changes, etc) is functional. ***IF*** you changed a password during any time and were not able to login to our new https://owa.xd.email/owa/ site, the password you set is likely in queue waiting to be applied to the mailbox. Once you can confirm you're good with OWA, Outlook should automatically offer to repair a profile and restart in the new one (no mail download, everything just starts working)

2. We are still working on reconnecting some PF for several organizations, this is done automatically on the backend. 

3. If you experience this issue with Outlook 2010:

Microsoft Outlook: There is a problem with the proxy server's security certificate.

Outlook is unable to connect to the proxy server cas.darkwing.exchangedefender.com. (Error Code 8000000)

Start > Control Panel > Mail > Show Profiles > Properties > Click on Repair

That will force Outlook to go out, check with Autodiscover, download the certificate, and apply it correctly. 

This only works on Outlook 2010. After that, the profiles automatically repair/redownload.

06/01/2020 16:41 PM 30 days

- Mail flow was resumed earlier today and all mail has been delivered to new Exchange mailboxes. There are roughly 100 domains that are still streaming mail in at 12:31 PM EST. For domains that have mail arriving since the cutoff, the delivery is in realtime.
- We are reconciling any orgs that are not in 2016 in the right place in the Support Portal under Exchange Service Manager.
- We are still going through password changes and login details. Most accounts we are going through have no problem logging in to https://owa.xd.email/owa/ so please if you can rely on that at the moment.

06/01/2020 15:07 PM 30 days

Mail flow to all our Exchange clients has been restored, along with distribution groups, shared mailboxes, etc. We have a few more tasks but all our Exchange clients should be in production.

We have received reports from some users that their passwords aren't working in Outlook. If you are experiencing this problem, please update the ticket with the list of email accounts (UPN) that are having an issue and we will start troubleshooting them. For the time being, if you are experiencing login issues please keep users in LiveArchive at https://nge.exchangedefender.com and OWA at https://owa.xd.email/owa/ (the trailing slash at the end is required, just typing in owa.xd.email will not work for now)

P.S. NO configuration (DNS/etc) change is required, this process is largely transparent. At most, you will have to click on OK/Rebuild/Restart when Outlook attempts to open the mailbox on the new infrastructure.

06/01/2020 13:22 PM 30 days

We are in the final stage of the checkdown, reviewing the Public Folder and Distribution group transition. We're working as quickly as possible and look forward to restoring Outlook service momentarily, please keep your users in LiveArchive for the time being (it works on mobile phones too)

06/01/2020 11:14 AM 30 days

As of 7AM the migration has been completed and we are nearing the end of all the error checks with the hope of restoring mail flow and Outlook connectivity shortly.  We are still working on a few items (public folders, changed UPN accounts). 

05/31/2020 23:12 PM 30 days

We are currently in the final stage of finalizing moves from our Exchange legacy platforms/clusters to the new 365/2016/2019 SKU. The work is expected to be completed before midnight, May 31st, 2020. While we do not anticipate any issues and have tested everything thorouhgly, things in IT (and legacy apps) can cause problems and we've stepped up our staffing to help our partners and clients work through them all. 

In the meantime, if your mail flow or access to Outlook gets interrupted, you are not down. There are two failover systems at ExchangeDefender you can rely on to continue working:

    ExchangeDefender LiveArchive: https://nge.exchangedefender.com 

    Outlook Web App: https://owa.xd.email/owa

We will keep you updated on our progress here at www.AnythingDown.com and at https://support.ownwebnow.com 

ExchangeDefender Database Maintenance (Resolved) Issue / Performance
12 days

At approximately 11:45 AM EST we were forced to restart our primary database clusters for an urgent performance issue that could not wait until after hours.

If you received "504 Gatweay Timeout" error from one of our web services it had to do with the service restart. As of 11:52 AM everything is back to normal.

smart host access from O365 (Resolved) Issue / Performance
12 days

We have been working with Microsoft today to address an apparent outage. The firewall and access rules have been reconfigured and our tests at this moment suggest that the issue has been resolved and mail is no longer being rejected as of 5:30 PM.

Our systems were processing the messages on behalf of our clients correctly and delivering them to the correct Microsoft endpoint, where connection was authenticated and recipient/detail exchanged. At that point Microsoft issued a 4.x.x temporary failure error ("Server busy. Try again later..") and ExchangeDefender continued making delivery attempts with some Microsoft servers issuing a 4.x.x error, some rejecting it with a 5.x.x, and some processing messages correctly. We have been in touch with Microsoft throughout the day and were able to have it addressed. As of 5:30 PM there doesn't appear to be an issue.

r.xdref.com Certificate Expired (Resolved) Issue / Performance
13 days

The certificate has been renewed and applied to the load balancer.

05/18/2020 12:17 PM 13 days

r.xdref.com is showing an Expired Certificate.

We are in the process of correcting this.

We are sorry for the inconvenience.

Swapping back to primary firewall (Resolved) Investigating / Notice
13 days

Access to all services is working according to our testing. We have a few things to wrap up but the fixed upgrade from the vendor appears to be working as intended 

05/16/2020 03:20 AM 15 days

The swap has completed but some services are still inaccessible. We're working on diagnosing the issue.

05/16/2020 01:44 AM 15 days

We will be setting our primary firewall back into active service which is to reverse the activation of the backup firewall

From 11:00 PM - 11:10 PM Eastern service access will be intermittent while the connections stabilize  

We are seeing an uptick in login failures and we are investigating. (Resolved) Investigating / Notice
16 days

This has been resolved, and reportedly affected mailboxes are successfully logged in at this time.

05/15/2020 17:38 PM 16 days

We are currently investigating an error some mailboxes are giving at login:

Your mailbox appears to be unavailable. Try to access it again in 10 seconds. If you see this error again, contact your helpdesk.


Users can be directed to use LiveArchive as a workaround in the meantime, by signing into their account here: https://admin.exchangedefender.com/livearchive.php

Thank you for your patience and understanding during this process. 



Hotfix update to firewall (Resolved) Investigating / Notice
13 days

The update was successful, but appears to not resolve the issue with a previous update. We will be activating our backup firewall to allow our vendor to further diagnose the issue

05/06/2020 16:57 PM 25 days

We've received an emergency hotfix from our firewall vendors to apply to all edge controllers. We will be applying the hotfix tonight at 11PM Eastern and rebooting the ingress routers. During the install all inbound services will be interrupted until the reboot completes. We anticipate the update taking up to 10 minutes to apply. 

Expired certificate ROCKERDUCK (Resolved) Issue / Performance
8 days

certificate updated at 4:23PM Eastern

04/23/2020 20:19 PM 7 days

The SSL for rockerduck has expired and was not swapped out. The certificate has already been renewed and is being distributed to the front end nodes

Database rollback (Resolved) Issue / Performance
10 days

During a schema upgrade dry run, a failure was detected which caused our monitoring software to rollback the latest backup on disk. This has unfortunately reverted changed from the past 48 hours meaning additions and changes to exchangedefender domains, users etc have rolled back. We are pulling the offsite copy from this morning and are working to restore the correct data structure.


Mobile Email Connectivity (Resolved) Investigating / Notice
18 days

As of 10:07 EST everything is operating normally and there have not been any new reports of an issue.

04/13/2020 12:33 PM 18 days

Mobile email connectivity is reported to be impacted for users on Louie. We are actively investigating the issue.

GLADSTONE offline (Resolved) Outage
25 days

Service was restored around 2:13 PM Eastern

At 1:51 the OS on the primary mailbox server forced a restart. At the time of the reboot, the replicas were 8 minutes behind and could not automatically fail over without force. Upon checking the crashed server, it was already in the process of loading into windows. We elected not to force a fail over as we did not want to lose the pending 8 minutes of data. Prior to the reboot, we had reports from various partners regarding IOS connectivity and OWA connectivity. We're still monitoring the situation, but at the moment all services are reporting operational.

04/06/2020 17:56 PM 25 days

Our monitoring has notified us about access issues to GLADSTONE starting at 1:53 PM Eastern. We are actively investigating the issue.

Mobile Email Connectivity (Resolved) Investigating / Notice
25 days

Mobile email connectivity was impacted for users on Gladstone and our backend team has made an adjustment on the server and expect mailflow to return to normal on mobile devices for these users.

If you have users reporting issues from earlier today and have not yet opened a ticket we request that you please have your users try again as this is reported as corrected and first users are reporting fully restored functionality.

Thank you for your patience and understanding at this time.


Network latency (Resolved) Issue / Performance
3 hrs

Network access and latency issues were resolved within minutes of this NOC alert being opened. We have resolved the issue temporarily and will put in a more permanent fix in the off peak hours during maintenance cycle.

03/31/2020 15:52 PM 2 hrs

We are noticing increases in request latency for bot web UI portals and mail flow. We're currently investigating the cause.

Primary firewall firmware update (Resolved) Investigating / Notice
6 hrs

At 11PM Eastern we will be performing an emergency firmware upgrade to our core ingress firewall. During the upgrade, all services will be temporarily interrupted for five minutes. If we run into any issues after the upgrade, we will activate our backup ingress firewall which is running the current configuration and firmware. 

ExchangeDefender Network Performance (Resolved) Issue / Performance
10 days

We are working on several network services today as we anticipate the growth in usage across all our services. We are currently experiencing degraded performance on our database servers which is slowing down and timing out some services.  More capacity is being onboarded, we appreciate your patience with us while everything syncs up.

Exchange 2016 Performance (Resolved) Issue / Performance
12 days

Service was fully restored around 11AM Eastern with no further issues reported by our partners

03/19/2020 14:15 PM 12 days

We are actively working to resolve intermittent access issues to 2016 mailboxes. Mobile phones seem to be the least impacted, followed by Outlook and then OWA. We anticipate all connections stabilizing by 11AM Eastern but customers should see performance increases as we update throttling policies.

Web UI Latency (Resolved) Issue / Performance
12 days

We've implemented our work around solution and we're seeing dramatic improvements in response time. We will continue to monitor the performance and work on fixing the underlaying issue with mail routing through LAX. At the moment, there are no observed delays

03/11/2020 13:46 PM 20 days

We've reconfigured our LAX data center to proxy as much mail traffic through Dallas to alleviate any mail delays that were observed yesterday as we continue to repair an issue in LAX. Unfortunately the increased load in Dallas is causing occasional timeouts on sites like support.ownwebnow.com, admin.exchangedefender.com, etc. We are close to implementing another workaround solution which should resolve the mail delays from yesterday without impacting the performance in Dallas

Potential mail delays (Resolved) Investigating / Notice
12 days

We've implemented our work around solution and we're seeing dramatic improvements in response time. We will continue to monitor the performance and work on fixing the underlaying issue with mail routing through LAX. At the moment, there are no observed delays

03/10/2020 14:35 PM 21 days

We've re-enabled our LAX datacenter and we are processing mail in both locations. We will monitor the load distribution over the next 15 minutes and update this NOC when service is fully operational

03/10/2020 14:18 PM 21 days

We're working on an issue with our secondary datacenter. For the time being, we've disabled mail from being accepted by our LAX and routing everything through our primary data center. We haven't seen any spikes in delays, but there is potential for 5-10 minute delays. We will update this post when all data centers are back to full operational status.

Incomplete SMTP mail logs (Resolved) Investigating / Notice
5 days

Our LAX cluster suffered a failure when logging SMTP transaction logs for search in the web UI for messages that arrived in the past 24 hours. Users will still be able to search and view the meta data of a message (from, to , subject, timestamp) but will not see the actual delivery logs if the message was processed by our LAX data center.


If you require the logs for a specific message, please open a support request with the meta information of the message and our staff will pull the logs from the backend.

Mailbox failover in Exchange 2016 (Resolved) Outage
6 days

As of 3:51 all users are confirmed as failed over and operations are back to normal. We will continue to work with Microsoft to ensure this bug / issue is properly diagnosed and leads to a more permanent solution. 

02/25/2020 20:38 PM 5 days

Service restart has forced the mailbox instances to start failing over. We should have services fully recovered for all users shortly.

02/25/2020 20:24 PM 5 days

We are experiencing a minor outage affecting 84 mailboxes on our Exchange 2016 platform. Our teams are trying to determine why the failover did not occur correctly, and out of abundance of caution, we aren't terminating the system call.

In the meantime, all 84 users affected by this Exchange database outage can still send and receive email at https://livearchive.exchangedefender.com while the maintenance is underway.

Intermittent outages on 2016 (Resolved) Issue / Performance
7 days

All work was completed successfully and things have been moving smoothly since Saturday morning. The new configuration was tested and failover is now behaving as intended. 

02/21/2020 06:50 AM 10 days

We successfully replaced the core router with a spare that we had on hand. We will still replace the unit with the shipped replacement from Cisco Friday or Saturday evening. This NOC will be updated once the swap out is completed.

02/21/2020 05:49 AM 10 days

We will be performing the hardware swap out in 10 minutes. Service is expected to be back online by 1:15AM Eastern

02/21/2020 05:11 AM 10 days

We're in the process of reprogramming one of our spare Cisco routers in order to temporarily replace the current master. We've elected to go this route to allow us to flip back to the original master without a conflict of configurations and addresses. 

We will attempt to activate the spare router in one hour at 1AM Eastern. During the swap, users will not be able to access their mailbox. We've allocated a 45 minute window to perform the swap. If we are unable to complete the swap in the time frame we will revert the changes and continue to work on a temporary solution.

02/20/2020 23:03 PM 10 days

Our team has been working on this issue all day and after consulting with Cisco we are going to attempt a temporary solution that will hopefully minimize the impact to the 2016 infrastructure until the routers can be managed on Saturday, during our weekly maintenance interval where we can minimize impact and continue to serve our clients with minimal interruptions.

We understand this is a serious issue as any impact to performance is an impact to the productivity of the people we serve. We have been fortunate that Outlook is fairly resiliant and have only had a few complaints. Mail isn't bouncing, and disconnections only last from a few moments to a few minutes and happen every few hours. We will attempt a workaround at 2AM EST and hope to be able to restore performance to normal at least until Saturday when full maintenance cycle can be performed. In plain English: we're trying to patch it along so it can make it to the weekend where we can sustain a potential outage of 30 minutes.

02/20/2020 15:33 PM 11 days

We've identified an issue with a core ingress router to our Exchange 2016 network. We will replace the router during our core maintenance cycle this weekend. During the router replacement, we will actively keep connections from opening while we validate the performance and stability of the replacement router. We'll update this posting during the upgrade.

Dallas Routing Issues (Resolved) Outage
18 days

At 8:12AM our Dallas DC firewall infrastructure suffered a catastrophic failure and failover wasn't instant across all services. Failure in one of the firewalls disconnected several private VPN connections that isolate/separate and protect the ExchangeDefender's internal services. We were forced to restart firewalls, which had to reestablish VPN connections, which required all the services to reload configurations simultaneously as routing work was started to move traffic around and away from Dallas. At about 8:24AM all services have been confirmed as green and back in services.

We have double checked every system and everything is back to normal.

Outage lasted approximately 10 minutes.

02/13/2020 13:39 PM 18 days

Private VPN tunnels are back up and all the log, maillog, ips, services are verified to be back in sync.

02/13/2020 13:30 PM 18 days

As of 8:21 the routing and almost all the applications have been restored. We are still working on what caused the problem and how to mitigate it automatically. 

02/13/2020 13:20 PM 18 days

We are working on a massive outage in Dallas. Service outages have been reported, we are working on it and will provide updates every 15 minutes as they become available.

ExchangeDefender inbound mail delay (Resolved) Issue / Performance
24 days

As of 11AM, the messages that were delayed in our Dallas DC have been processed and delivered. All other data centers processed mail without an issue.

02/07/2020 15:51 PM 24 days

As of a few minutes ago all the new mail is going through the load balancers without delay, the mail that was delivered to nodes that were experiencing DNS/latency issues in Dallas is being processed at the moment and will be delivered in a few minutes.

ExchangeDefender database upgrade (Resolved) Investigating / Notice
12 days

We will be upgrading to a new database cluster at 4 PM EST on 2/6/20

During the upgrade all web UI services will be taken offline for 5 minutes. The services affected will be

  • Admin UI
  • Encryption
  • Web file share
  • Live archive

Mail flow will not be interrupted at all during the upgrade

Exchange 2016 clock drift (Resolved) Issue / Performance
2 days

The issue with the clock drift has been addressed and time is displaying correctly on new messages. No mail was lost or delayed during this issue but some messages got a timestamp of up to 5 minutes in the future.

01/29/2020 16:45 PM 2 days

We've been addressing an issue with clock drift on one of our Exchange 2016 nodes. The drift is less than 5 minutes and should be resolved momentarily. 

SPAM targeting Invoicely (Resolved) Investigating / Notice
12 days

We are currently tracking large amounts of messages from recently registered .site domains that are forging their messages to appear as Invoicely. You may see subjects like the ones below, if some are released to your Inbox please delete them:

Past-due payment notification from Invoicely

Past due invoice notification from Invoicely

Overdue invoice notification from Invoicely

Delivery delays and issues with btinternet (Resolved) Investigating / Notice
4 days

BT Internet DNS issue seems to have been addressed. We have no visibility into their network, however, one company has let us know that several items sent to BT did not make it to recipients mailboxes even though the delivery has been confirmed. Unfortunately, free email accounts and ISP email accounts tend to do this from time to time, the only option is to resend.

01/24/2020 17:41 PM 7 days

We are currently working with British Telecom to address an issue on their servers related to their implementation of  synchronoss.net software. What our customers are experiencing, intermittently, are delivery delays while sending messages to btinternet and associated properties. The issue was first reported to us around 9AM EST and was quickly addressed to bypass/mask the ExchangeDefender network and route around the problem until BT has a chance to figure out why their DNS and outsourced mail service are having issues with DNS resolution.

Current error produced on their side is:

421 re-prd-rgin-021.btmx-prd.synchronoss.net Service not available - no PTR record for x.x.x.x


As of 10:15 AM this morning this issue has been addressed for our customers so they can send mail to BT, however, this is an issue outside of ExchangeDefender's control and free/ISP mail staff tend not to be very concerned about email delivery issues in general so we do not expect a quick resolution nor can we estimate an ETA. 

Performance of r.xdref.com (Resolved) Issue / Performance
7 days

The issue that caused performance impact and timeouts has been addressed overnight on the backend. We have improved the failover and load balancing of our container infrastructure that handles services that were impacted in our phishing.

01/23/2020 18:05 PM 7 days

ExchangeDefender SQL driven properties (admin, r.xdref.com, etc) are moving slower than usual today, we are allocating additional resources to the instances serving these services and expect the performance to return to nominal by 1:20 PM EST. The issue is within the ExchangeDefender cloud and is being addressed by our team right now.

Large scale phishing attack from Russia (Resolved) Investigating / Notice
12 days

We are presently experiencing a massive phishing attack originating in Russia targeting enterprise users from Office365 and beyond. All messages suggest that the invoice/order/bill/payment/transaction has been completed and encourages clients to click on a link that either has a dangerous malware payload or identity theft attempt. This is similar to the Azure and Amazon Web Services (aws) outbreak that has been gaining momentum for the past week.

If you have ExchangeDefender, you don't have anything to worry about - we're blocking their access to our clients and anything that does slip will be picked up by the ExchangeDefender Phishing Firewall which will block traffic to those dangerous sites.  

If your ExchangeDefender service is managed by a third party (MSP) please check with them directly to see if they have blocked ExchangeDefender Phishing Firewall because that action would put you at risk. 

Public folders offline (Resolved) Issue / Performance
22 days

Public folders are now online in the ROCKERDUCK cluster

01/09/2020 13:54 PM 22 days

We are working to restore access to the primary public folder database in the ROCKERDUCK cluster. Currently, users are unable to access public folders through all mediums. We anticipate restoring service within 20 minutes.

Dismounted database ROCKERDUCK (Resolved) Issue / Performance
23 days

We received an alert about a single dismounted database. The original hosting server is indicating performance issues which has prompted us to activate a switchover for the database. We are working to restore service to the users on the affected database.

Live Archive access issue (Resolved) Issue / Performance
1 hrs

We have resolved issues with the proxy/redirection/caching that affected some LiveArchive logins. The LiveArchive login now inherits the functionality/style of all the other ExchangeDefender/Wrkoo products.

12/31/2019 20:26 PM 30 days

Some users are having issues accessing their Live Archive accounts, we are working on a solution. We are sorry for the inconvenience this may have caused.

SSL certificate issue on scrooge cluster (Resolved) Investigating / Notice
10 days

We are currently investigating an issue with a certificate chain on one of the Exchange CAS nodes in the Scrooge infrastructure. If you've been disconnected your Outlook will attempt to reconnect automatically after a few minutes. 

LOUIE access issue (Resolved) Issue / Performance
18 days

We are seeing failed requests for 13% of connections to LOUIE. We're currently investigating the issue

Office 365 Outbound issue (Resolved) Issue / Performance
27 days

The issue should be fixed.

Please verify. Thank you.

12/04/2019 14:27 PM 27 days

Microsoft is currently making changes to their new servers. It appears the 365 team is using *.prod.outlook.com for newer servers vs the previous rDNS name of *.outbound.protection.outlook.com . We have added the new host name to the allowed list which should fix it.

We will monitor the situation. Thank you!

Malwarebytes false positive (Resolved) Investigating / Notice
27 days

Malwarebytes have confirmed that the issue has been fixed and the site is whitelisted again.


Thank you for your patience.

12/04/2019 14:24 PM 27 days

We are currently receiving notifications of a false positives for our support site.

Please ignore this as it  is a false positive. We are working on getting this removed. Thank you.

POP3/IMAP/Webmail for Webhosted Mail Inaccessible (Resolved) Outage
20 days

We've opened this issue again as there are still issues accessing services. We've temporarily disabled POP and IMAP connections to provide at least a reduced consistent connection

11/22/2019 15:38 PM 9 days

Service has been restored and mail is flowing. The original and replacement motherboard were both defective but remedied by a replacement motherboard that was overnighted to our datacenter.

11/21/2019 16:49 PM 10 days

We received alerts about the mail service included with webhosting being offline. We are beginning our investigation and will add additional notes to this NOC as progress is made.

Fixed XDPF Rewriting for multiple hashtags in URL (Resolved) Investigating / Notice
10 days

On Tuesday, November 19th, 2019 we addressed a bug in the ExchangeDefender Phishing Firewall URL rewriting software that was improperly truncating URLs. Our partners at ReadyCrest Ltd identified the bug by comparing the length of the compressed, rewritten and raw URL and seeing that we truncate link after the first #.Usage of # tends to be fairly explicit in HTML emails and almost exclusively used for anchor links. These links don't actually connect to another site/document, they are meant to scroll the browser to a specific part in the web page. Typically they are used in a table of contents for longer pages, contracts, etc to move the browser to where the actual content is. The syntax is # followed by the name of the section to scroll to - so there is NO functional reason in HTML to ever have more than one # in the URL. That was key in our design and decision to drop anything past the second # - but we've now discovered that some sites may, in fact, put extra (typically reserved) special characters like # in the URL which breaks ExchangeDefender Phishing Firewall rewrites because it removes portions of the link.To our knowledge, this is the first reported link with multiple # in the URL, and going forward multiple # will be allowed in the URL without truncation.This issue has been addressed.

LOUIE access issues (Resolved) Outage
18 days

Access has been restored to users on the dismounted database. We are now investigating why the automatic failover didn't occur for the dismounted database.

11/13/2019 01:12 AM 18 days

A mailbox node in the LOUIE cluster lost one of its uplinks and has caused one database to become dismounted. Prior to losing the uplink, the node held 6 mailbox databases and five successfully failed over to another node. It is currently unclear why the single database did not fail over to a replicated node, but we will investigate it further after we've restored service. We're in the process of replacing the uplink and service should be restored shortly.

Encryption Portal (Resolved) Issue / Performance
18 days

We are currently investigating the Encryption Portal and some users reporting issues with opening emails and seeing the subject from the Encryption Portal.

Please standby as we take a deeper look into this. Thank you!

Incorrect Invoice Subject (Resolved) Investigating / Notice
19 days

This issue has been resolved. Thank you!

10/31/2019 22:18 PM 30 days

The invoices sent to our clients for November came with the subject "Monthly Services (December, 2019)" instead of November, 2019. We've made appropriate adjustments on the backend and everything is labeled correctly as of 6PM EST, October 31, 2019.

Service Portal peformance (Resolved) Issue / Performance
19 days

This issue has been resolved. Thank you!

10/29/2019 15:37 PM 2 days

We are currently investigating issues with portals in regards to our services.

Please stand by. Thank you.

SMTP log issue (Resolved) Issue / Performance
3 days

The issue has been fixed.

All new email will reflect properly in the logs.

Thank you for your patience.

10/28/2019 17:23 PM 3 days

We are currently expecting some issues for SMTP logs for some domains. Logs are showing blank for a few domains. 

Please standby as we fix this. Thank you!

SharePoint Performance (Louie) (Resolved) Issue / Performance
7 days

This issue has been resolved. All operations are back to normal.

Thank you.

10/24/2019 14:05 PM 7 days

We are experiencing further issues on this specific server with SharePoint.

We will update once we have a resolution. Please stand by!

Thank you.

Preview infrastructure (Resolved) Issue / Performance
9 days

The issues were identified fixed with the servers.

Thank you for your patience in this matter! 

10/22/2019 15:12 PM 9 days

We are investigating an issue with our message preview infrastructure, possibly affecting a few users on Encryption, Compliance Archiving, and SPAM Quarantines. The issue should be resolved momentarily if you have an issue with a preview loading just hit refresh.

Thank you!

SharePoint Performance (Resolved) Issue / Performance
9 days

The issues were identified fixed with the servers.

Thank you for your patience in this matter! 

10/22/2019 12:26 PM 9 days

We are getting reports of certain domains not being able to log into their SharePoint sites.

We are investigating the issue and will keep you updated.

Thank you for your patience.

Accelerated update to inbound load balancers (Resolved) Issue / Performance
14 days

We've completed an upgrade to our load balancing infrastructure. These servers and switches take on the traffic from the Internet and pass it on to appropriate inbound nodes in the ExchangeDefender cloud for processing. While we have been hoping to roll out the upgrade to the load balancing infrastructure later in November during the long Thanksgiving holiday, we simply needed next-gen stuff immediately. The event that occurred on Wednesday, October 16th, 2019, would have been fully and immediately mitigated by the new load balancers so we are hoping that this upgrade starts performing well for our users immediately. We activated the new load balancer in our primary location (Dallas) at midnight on October 17th and immediately saw a performance gain on the processing speed for inbound messages.

Admin page (Resolved) Issue / Performance
27 days

The issue at hand has been fixed.

Please verify that the page is working properly.

Thank you for your patience.

10/04/2019 13:56 PM 27 days

It has been brought to our attention that there is some user currently being affected by performance issues on https://admin.exchangedefender.com/

We are currently investigating the issue.

Thank you for your patience. Please standby.

SSL refresh for Exchange 2016 (Resolved) Investigating / Notice
13 days

The certs have been refreshed

09/17/2019 14:06 PM 14 days

Tonight at 11PM Eastern we will be restarting the load balancer service for 2016 to use the renewed SSL for the root domains. Access should only be interrupted from 11:00 PM - 11:05 PM Eastern.

SSL xdref.com (Resolved) Issue / Performance
2 days

Thank you for your patience.

The Engineering team has adviced that the issue has been fixed.

Please have all users affected by this and verify. If you are still facing issues please open up a ticket on Support Portal, please.

Thank you.

08/29/2019 11:18 AM 2 days

We are currently experiencing some SSL Certificate issues with xdref.com. We have identified the issue and are currently working to resolve it. 

In the meantime have users Click on Advance and  select Proceed to r.xdref.com

This is so they can still keep going to their links.

We appreciate your patience.

Automatic Provisioning of ExchangeDefender - Reports Investigating / Notice
4 days

This NOC alert is regarding a timely / deadline-driven issue related to billing:

This month we launched the replacement to XD Sync and it has been working really well in terms of identifying email addresses that are sending mail but do not have valid accounts in ExchangeDefender.

You can read more about it here: http://www.exchangedefender.com/blog/2019/08/exchangedefender-account-provisioning-live/

What we are adding at the moment, and will have ready by EOD August 28th, 2019, is a centralized report and visual indicators for Service Provider/CIO personnel to see which accounts have been added so that there is no surprise on the 1st of the month when invoices go out.

Information will be posted on our blog at https://www.exchangedefender.com/blog

Encryption Portal Error 1 (Resolved) Issue / Performance
4 days

Thank you for your patience.

The developers have advised that this issue has been fixed. It was due to a small bug that was easily and quickly repaired.

Please have any users previously affected by this issue try again with reading any messages.

Thank you.

08/27/2019 19:29 PM 3 days

Some users are experiencing issues with the Encryption portal (Error 1).

We are currently investigating. 

Please standby.

ROCKERDUCK Public Folders (Resolved) Outage
9 days

Access has been restored to public folders.

08/22/2019 16:03 PM 9 days

Customers with public folders are unable to access public folder data. We're working on restoring access as quickly as possible

Della peformance (Resolved) Issue / Performance
12 days

Thank you for your patience in this matter.

The issue at hand has been fixed. If you are still experiencing issues please open up a ticket on here.

Thank you.

08/16/2019 16:45 PM 15 days

We are currently facing some issues with the Della Server.

Please have users use LiveArchive while we pinpoint this issue.

Thank you for your patience.

Autodiscover prompt on 2016 (Resolved) Investigating / Notice
12 days

Thank you for your patience in this matter.

The issue at hand has been fixed. If you are still experiencing issues please open up a ticket on here.

Thank you.

08/15/2019 14:05 PM 16 days

Between the hours of 4am - 930am Eastern, Outlook users would have seen a autodiscover prompt for an account not tied to their organization (billing@travispot.com) in error. The autodiscover prompt was a result of us testing public folders for our 2016 environment and that account was one of our demo public folders. Although no users could see or open the public folder, autodiscover detected it as an available mailbox and begun to prompt users about allowing Outlook to try and connect to the public folder.


The public folder was disabled and we will investigate this further into the off hours

Della Issues (Resolved) Issue / Performance
2 days

We have fixed the issue with Della.

The issue was attributed to some settings that were promptly enhanced in order to prevent this issue in the future.

Please verify and thank you for your patience.

07/26/2019 13:33 PM 5 days

We are currently facing some issues with the Della Server.

Please have users use LiveArchive while we pinpoint this issue.

Thank you for your patience.

Admin Portal/XDREF Feature Update. Investigating / Notice
20 days

New features/Options are now available  in the Admin Portal:

- Ability to upload a file (csv) that contains multiple blacklist entries (domain level)

- Phishing logs (sp,domain, user level )

XDREF Feature Update

Added - When whitelisting or blacklisting, XDREF will now be adding the same rules to any alias's you may have. This includes alias's domains as well (if you're whitelisting on the Domain level) 

r.xdref.com Feature Update Investigating / Notice
21 days

Quick update, we continue to work on r.xdref.com as we roll out new functionality, settings, and ease of use. Here is what we did today:


- Currently working on the ability that if you whitelist or blacklist for one alias, it'll trickle down to all aliases and the main account. 

- To add to that, the functionality to whitelist and blacklist items for an alias domain to trickle down to all alias domain's and the Main Domain.

- Rolled out the wild card feature we have on our standard whitelists and blacklists. Meaning you can now do an entry for :

- *.domain.com - to grab all sub-domains for the domain

- domain.com - to whitelist/blacklist the domain itself

We'll continue to update you with new features as we continue to update the service. 

Della peformance (Resolved) Issue / Performance
21 days

The issues with DELLA is fixed.

We have verified that email is flowing properly.

Again, thank you for your patience in this matter.

07/10/2019 11:32 AM 21 days

We are currently seeing some issues with DELLA.

We are looking into the issue and  are working on getting it resolve ASAP.

Please standby and thank you for your patience on this issue.

r.xdref.com Update Investigating / Notice
22 days

Quick update, we continue to work on r.xdref.com as we roll out new functionality, settings, and ease of use. Here is what we did today:

- Reduced log databases, seeded archive log databases

- Rolled out analytics modules for frequent refresh (redirect to help in case they aren't reading the screen)

- Added more tld validations to make sure hackers can't sabotage the firewall by whitelisting all tlds (our liability)

- Exposed additional monitoring and management control panels for support for immediate remidiation

- Added additional warnings for URLs (whitelisted or not) that lead to executable content

ExchangeDefender xdref.com Phishing UI Changes Investigating / Notice
23 days

We have been rolling out new features and fixing bugs / tweaking things on r.xdref.com all day, here is the summary so far:

- We have made the Link: window more prominent

- We have added a link to Report Issue for phishing advice & support purposes.

- We have been optimizing the search and redirect code all day, to automatically review/whitelist sites that are visited often.

- BUGFIX: Whitelist core

- BUGFIX: Email redirection, mailto: automatically redirects

- BUGFIX: Redirect on Whitelist, when client clicks to whitelist and is authenticated, system will automatically redirect

- BUGFIX: Change text from "Link:" to "Click below to proceed to the site:"

ExchangeDefender xdref.com Pishing Firewall Whitelists (Resolved) Issue / Performance
26 days

During the final phase of ExchangeDefender Phishing Firewall deployment, our whitelists briefly stopped working. This means that the users were prompted with the info screen every time they clicked on any link, good or bad, which should not be happening. An emergency patch at 2:30 PM EST was deployed and now whitelist and permitted list code is in effect.

Info: By default, ExchangeDefender Pro clients will only see the warning screen under 2 conditions: 1) Domain they clicked on is categorized as a malicious source of phishing/malware or 2) Domain is not one of the top domains on the Internet and is not on the whitelist (typically, this is where phishing content is generally directed to). For most users, warning/redirection screen should seldom be displayed.

SharePoint errors (Resolved) Issue / Performance
3 days

The issues was identified as the servers were not able to boot up after hardware swap for improved performance.

This was quickly fixed and the performance of the server was enhanced.


Thank you for your patience.

06/28/2019 12:43 PM 3 days

We are getting reports of certain domains not being able to log into their sharepoint sites.

We are investigating the issue and will keep you updated.

Thank you for your patience.

General Internet Outage (Resolved) Investigating / Notice
6 days

The outage seems to be reported as fix about 16:00 EST yesterday.

It seems Verizon was responsible for the outage due to a widespread BGP routing leak. This affected a number of Internet services and a portion of their traffic.

BGP acts as the backbone of the Internet, routing traffic through Internet transit providers and then to services. There are more than 700k routes across the Internet. By nature, route leaks are localized and can be caused by error or through malicious intent.

Thank you for your patience in this matter.

06/24/2019 13:52 PM 7 days

It has been brought to our attention that several popular sites like Google, Amazon, Reddit, and Spectrum — just to name a few — are experiencing issues this morning. Those issues appear to have begun around 6 or 7 AM ET.


Problems appear to be clearing up as of 8:40 AM, it's likely going to take some time before everything is running smoothly again. Reports are still going up on DownDetector as of writing.


We will continue to monitor the situation as it develops.

Thank you for you patience in this matter.

Admin Maintenance (Resolved) Issue / Performance
11 days

If your users haven't received their scheduled Quarantine Reports, please know that you can resend the reports. 


Simply log into (https://admin.exchangedefender.com) with either your SP or Domain level account.

Once there, select accounts, and find the users who are reporting that they haven't received it. Once they've been selected, click on the Actions button then click on the "Resend SPAM Report" Button. 


If you're still experiencing issues with receiving SPAM Messages, please let us know and we'll be more than happy to take look into it. 


06/20/2019 15:00 PM 11 days

The issue with the admin site (https://admin.exchangedefender.com) and ExchangeDefender Encryption portal (https://encryption.exchangedefender.com) has been addressed. The issue was related to some services exceeding connection counts (similar to "all circuits are busy now") in which some clients were displayed the error when the connection to the backend services could not be made. Out of abundance of caution, we put up a maintenance notice while resource adjustments were done. Everything is back to normal, mail is flowing.

Specifically, the issue was with the database facilities used for event logging. It did not affect the actual mail processing services.

503 Phishing Links (Resolved) Issue / Performance
14 days

The issue has been fixed. It was found out to be an issue with our hosting company.

Thank you for your patience.

06/17/2019 13:26 PM 14 days

We are currently getting error reports about phishing links redirecting to an error 503.

We are currently working on fixing the issue.

Warning Messages (Resolved) Issue / Performance
14 days

As of noon EST, the issue has been addressed and we have not seen another instance of a malware related DDoS. The problem has been addressed around 8:30 EST, fixed, put back into production for testing at 11AM and into full production around 11:30. As of 2PM EST, everything is in the clear.

For the messages that were received previously and that were affected by the DDoS, the malware scan only disarms the dangerous HTML entities. All legitimate email clients send messages in both HTML and text content, and they will show HTML version if available by default (hence why some saw the warning). To see the actual message just right click, select View Source, and you'll see the text version of it.

As of 9AM, this bug affected 117 messages of which 82 came via typical "marketing" junk source from Indeed, Constant Contact, and vresp.

We believe this problem will not show up due to a change in how we're logging and disarming these bug/tracking entities, but we will keep an eye on it as usual.

06/14/2019 13:22 PM 17 days

We are still working on the issue that was causing the problem, and have disabled the scan that was being affected in order to assure email delivery.

We are investigating and will provide an update when the problem has been resolved.

However, all mail should be arriving without an issue and without the DDoS notice.

If the message had critical / important information (in our searches it appears to have overwhelmingly affected mass mailing platforms like indeedmail, constantcontact, vresp) you can still see it by viewing the source of the message.

Just Right click, select View Source and you'll see the text version of the message.

06/14/2019 12:49 PM 17 days


We are investigating a reported issue regarding DDoS warning messages this morning as we've had a higher than usual number of messages crashing the security/antivirus engine during the scan.


We hope to have it resolved shortly.

In the meantime, if you can ask the client to resend the message, please do.

Minor mail delays (Resolved) Issue / Performance
27 days

Mail flow speeds have returned to normal

06/04/2019 16:40 PM 27 days

We've enabled LAX and will be smoothing out the queues between the data centers for the next 15 minutes. Mail flow should resume normal speed by 1PM Eastern

06/04/2019 15:48 PM 27 days

Our monitoring begun alerting about latency and packet loss issues with our LAX data center starting at 11:31AM. To mitigate any potential issues with delivery, we've elected to pause mail flow going to our LAX datacenter. While LAX is paused, Dallas will see mail delays up to 5 minutes but is able to handle the overall load. We will enable LAX once we see the packet loss subside. 

SPAM Issues (Resolved) Issue / Performance
27 days

We have received reports of a large SPAM outbreak going through our Inbound network. 

The messages are of the sextortion variety and are claiming to have sensitive information/video/recordings of users and that they have to pay a Bitcoin ransom so that the perpetrators do not release the footage. 


Please do not fear as the messages are benign in nature. If you or your clients are currently receiving messages like that, please open a support ticket and attach the unforwarded (we need the ORIGINAL) message in the ticket so we can analyze the headers and send you what domains to blacklist. We'll also use the message you send as a sample to track the different variants of the messages so we can create rules and enforce them immediately.


We appreciate your patience in this matter. 

Issues with Della (Resolved) Issue / Performance
27 days

The issues with our DELLA Cluster has been resolved. 


One of the transport hub services wasn't running properly so we had to restart the service and mail began flowing properly. 


If you are still are experiencing issues with this, please let us know via a Support Ticket and we'll be more than happy to assist you. 


06/03/2019 12:07 PM 28 days

We are currently facing issues with Della server.
We are investigating and fixing the issue. 


Thank you for your patience.

Webfile Server Upgrade (Resolved) Investigating / Notice
27 days

Upgrade completed

05/30/2019 20:35 PM 21 hrs

At 9:00 PM Eastern we will be upgrading the Web File Server service in order to support larger downloads/uploads. The upgrade is planned to take less than 10 minutes to complete. During the upgrade, customers will receive a maintenance notification regarding WFS until the upgrade is completed.

Office power outage (Resolved) Outage
1 days

We have currently regained powered back at our main office.

We will monitor for 24-48 hours to verify that the issue has been fixed.

Thank you for your patience.

05/30/2019 11:37 AM 1 days

We are currently facing power outages at our main office. 


At the moment it's not affecting any mail flow and everything should be running as normal. 


We are monitoring the situation and will update ASAP.

05/29/2019 13:36 PM 2 days

We have currently regained powered back at our main office.

We will monitor for 24-48 hours to verify that the issue has been fixed.

Thank you for your patience.

05/29/2019 12:46 PM 2 days

We are currently facing power outages at our main office.


At the moment it's not affecting any mail flow and everything should be running as normal.


We are monitoring the situation and will update ASAP.

Replacing predicted failure disk (Resolved) Investigating / Notice
6 days

Completed without any problems.

05/23/2019 15:39 PM 8 days

On Friday evening between 9PM - 11PM we will be replacing a bad disk in the LOUIE cluster which currently holds public folder data and transport service data. During the maintenance mail flow will be uninterrupted for domains that have "cas.louie.exchangedefender.com" as their delivery point for ExchangeDefender. All public folder access will be disabled during the maintenance period, however, any queued mail for the public folders will remain in queue. Regular mailbox operations will be uninterrupted.

Emergency Maintenance on Inbound nodes 19,41 and 42 (Resolved) Issue / Performance
9 days

After monitoring the network overall during our peak EST and PST hours, everything appears to be back to our normal nominal network performance, and in many cases significantly better than even our ordinary levels considering the elevated mail flow.

05/22/2019 11:20 AM 9 days

ExchangeDefender staff continued to monitor and tune our load balancing infrastructure through our cloud, particularly in our Los Angeles data center. Upgrades, improvements, and other optimizations should significantly increase performance. We will continue to monitor the situation and update this post throughout the day as we have more information but mail delivery should be significantly faster today.

05/21/2019 21:25 PM 9 days

We believe that the issues in Los Angeles have been addressed but we'll continue monitoring these systems. Nodes in this cluster were pulled in and out of production periodically while we tried to diagnose performance related issues (antivirus, DNS, routing, etc).

05/20/2019 16:29 PM 11 days

We've discovered some performance issues in the following nodes:

-Inbound19

-Inbound41

-Inbound42

Our team is currently performing emergency maintenance on those nodes.

If you're experiencing any mail delays and the NDR's are saying they're coming from those nodes, please let us know. Our team is working hard to fix the issues so the delays do not continue., 


If you have any questions or concerns, please put in a support ticket @ https://support.ownwebnow.com or give us a call @877-546-0316


We appreciate your patience in this matter. 

Reverse DNS error (Resolved) Investigating / Notice
16 days

Issue has been resolved if any denials please try again. Thank you for your patience

05/15/2019 15:21 PM 16 days

We are currently looking into errors with Reverse DNS error for some users. We will update as it develops. Thank you.

Landlines (Resolved) Issue / Performance
16 days

Issue has been resolved. Thank you for your patience.

05/14/2019 19:43 PM 16 days

We are currently expecting issues with incoming calls to our landlines. We are working to fix the issue. We will update as it develops.

Level 3 outages (Resolved) Outage
15 days

The outages seem to be resolved. Thank you for your patience.

05/14/2019 19:41 PM 16 days

Currently Nationwide they are reported Level 3 outages. We will monitor the situation as it develops. Please expect slow response/loading times when using our system.

Sharepoint Issues (Resolved) Issue / Performance
14 days

Thank you for your patience. The issue has been resolved.

05/13/2019 15:26 PM 18 days

We are currently experiencing some issues with one of our Sharepoint nodes on Gladstone. 

Our team is currently investigating the problem and will have a resolution shortly. 

If you're on Gladstone and are experiencing issues with accessing Sharepoint, please open up a support ticket and let us know the sharepoint credentials so we can validate connectivity. 

Exchange Defender Quarantine Spam Reports (Resolved) Issue / Performance
22 days

The issue was fixed.

05/09/2019 12:54 PM 22 days

ExchangeDefender has received reports of several users receiving empty ExchangeDefender Quarantine SPAM reports. We are addressing the issue at the moment and will have it resolved today. In the meantime, all users can access their SPAM messages at https://admin.exchangedefender.com or by clicking on the link in the email.

ROCKERDUCK single DB dismounted (Resolved) Issue / Performance
23 days

We noticed an unusual high level of MAPI failed responses for a single database. We begun an active host switch over which should have immediately completed with no service interruption. Unfortunately, the mount didn't immediately finish and is still processing the switch over. This database holds 90 mailboxes. Users on this database will be unable to access their mailbox until the switch over is completed. 

POP3 server mail1 issue (Resolved) Outage
30 days

Service has been restored to mail1

05/01/2019 02:19 AM 30 days

We are investigating an outage on mail1.ownwebnow.com (mail1.exchangedefender.com). Currently all services are inaccessible. This affects some web hosting clients who use the POP services

Pausing mail flow in Dallas 9PM (Resolved) Investigating / Notice
30 days

This was completed without any issue

04/30/2019 17:11 PM 1 days

We will be restarting some Dallas ExchangeDefender inbound node core components at 9:45PM until 10:00 PM on 4/30/19. We will pause mail flow going to Dallas at 9:30PM to prevent any mail flow issues from occurring. Mail will continue to route through our other data centers and there will be no interruption in mail flow.

Inbound Phone Issues. (Resolved) Investigating / Notice
1 days

We are currently experiencing issues with our inbound phone system. If you require immediate assistance or a callback, please open a support ticket @https://support.ownwebnow.com and we'll be more than happy to assist you. 

We appreciate your patience in this mattter. 

Issues with Public folders on LOUIE (Resolved) Issue / Performance
7 days

The public folder issues on our LOUIE Cluster have been resolved. 

If you are still experiencing issues with accessing public folders, please let us know. 

We appreciate your patience in this matter. 

04/24/2019 14:57 PM 7 days

We are currently having issues with accessing Public Folders on our LOUIE Cluster. Our team is currently investigating these issues ans will provide an update as soon as it's been resolved. We appreciate your patience.

SSL Certificate Issues (Resolved) Issue / Performance
8 days

We have rectified the SSL issue on our Rockeduck cluster, we appreciate your patience in the matter. If you're still encountering issues, please call us 877-546-0316 or put in a support ticket https://support.ownwebnow.com. Thank you again for your patience.

04/23/2019 15:17 PM 8 days

We are currently experiencing some SSL Certificate issues on our ROCKERDUCK Cluster. We have identified the issue and are currently working to resolve it. We appreciate your patience.

Mail Delays (Resolved) Investigating / Notice
13 days

The issue with the Mail Delays across both the Dallas and Los Angeles Data Centers has been identified and repaired. Mail should start flowing normally within a few minutes. If you have any questions or concerns, please contact our support team via Ticket or call us at 877-546-0316. We appreciate your patience.

Exchange 2016 Database Distribution Rebalance (Resolved) Investigating / Notice
15 days

We have scheduled a rebalance of active hosted database distribution on our Exchange 2016 Nodes. This will have no impact on customer accessibility. 

LOUIEMBOX7 Performance Issues (Resolved) Issue / Performance
18 days

Reboots have been scheduled for LOUIEMBOX7 due to Memory issues. This is scheduled for 930PM EST today. This will have no impact on customer accessibility.

Comcast Blacklist Rejections (Resolved) Investigating / Notice
19 days

We have received two (2) client complaints about email being rejected by Comcast. 

At the moment we aren't seeing issues with delivery to Comcast, none of our IP addresses on any outbound networks are on an RBL (mxtoolbox.com), and manual telnet SMTP diagnostics tests are so far showing no issues either. This leads us to believe that it's a typical Comcast issue (rejecting mail randomly from different servers as they get swamped and overloaded during peak hours). Best practices for sending mail to Comcast or Yahoo is to just resend the message and hope a different server in their infrastructure accepts the message. When there are RBL issues, we immediately open a support request and start working with their postmaster staff.

Email delivery delays on Gladstone (Resolved) Issue / Performance
18 days

The issue was resolved and clients have reported no issues since.

04/11/2019 16:45 PM 20 days

Maintenance task has been completed and we have not seen nor have we received reports of any issues.

04/10/2019 14:00 PM 21 days

Some users on the ExchangeDefender Exchange cluster GLADSTONE experienced significant email delivery delays throughout the day as we rebalanced our storage. This is a one-time event that was a side effect of a major maintenance task required to facilitate a change in processing of our Compliance Archiving mail. Traditionally, we stored ExchangeDefender mailboxes and journal mailboxes on same volumes which during peak hours could create a performance issue, and this maintenance task assured that going forward Exchange bottlenecks will not be a source of issues for clients that remain on our legacy Exchange infrastructure.

If this issue affected your clients adversely, please schedule a migration of their mailboxes to Exchange 2016 with our team immediately and we will make sure that they are moved to the front of the queue.

ExchangeDefender LiveArchive still received and sent messages in realtime allowing our clients to not feel a major performance impact.