ExchangeDefender Database Maintenance (Resolved) Issue / Performance
5 days

At approximately 11:45 AM EST we were forced to restart our primary database clusters for an urgent performance issue that could not wait until after hours.

If you received "504 Gatweay Timeout" error from one of our web services it had to do with the service restart. As of 11:52 AM everything is back to normal.

smart host access from O365 (Resolved) Issue / Performance
5 days

We have been working with Microsoft today to address an apparent outage. The firewall and access rules have been reconfigured and our tests at this moment suggest that the issue has been resolved and mail is no longer being rejected as of 5:30 PM.

Our systems were processing the messages on behalf of our clients correctly and delivering them to the correct Microsoft endpoint, where connection was authenticated and recipient/detail exchanged. At that point Microsoft issued a 4.x.x temporary failure error ("Server busy. Try again later..") and ExchangeDefender continued making delivery attempts with some Microsoft servers issuing a 4.x.x error, some rejecting it with a 5.x.x, and some processing messages correctly. We have been in touch with Microsoft throughout the day and were able to have it addressed. As of 5:30 PM there doesn't appear to be an issue.

r.xdref.com Certificate Expired (Resolved) Issue / Performance
7 days

The certificate has been renewed and applied to the load balancer.

05/18/2020 12:17 PM 6 days

r.xdref.com is showing an Expired Certificate.

We are in the process of correcting this.

We are sorry for the inconvenience.

Swapping back to primary firewall (Resolved) Investigating / Notice
7 days

Access to all services is working according to our testing. We have a few things to wrap up but the fixed upgrade from the vendor appears to be working as intended 

05/16/2020 03:20 AM 9 days

The swap has completed but some services are still inaccessible. We're working on diagnosing the issue.

05/16/2020 01:44 AM 9 days

We will be setting our primary firewall back into active service which is to reverse the activation of the backup firewall

From 11:00 PM - 11:10 PM Eastern service access will be intermittent while the connections stabilize  

We are seeing an uptick in login failures and we are investigating. (Resolved) Investigating / Notice
9 days

This has been resolved, and reportedly affected mailboxes are successfully logged in at this time.

05/15/2020 17:38 PM 9 days

We are currently investigating an error some mailboxes are giving at login:

Your mailbox appears to be unavailable. Try to access it again in 10 seconds. If you see this error again, contact your helpdesk.


Users can be directed to use LiveArchive as a workaround in the meantime, by signing into their account here: https://admin.exchangedefender.com/livearchive.php

Thank you for your patience and understanding during this process. 



Hotfix update to firewall (Resolved) Investigating / Notice
7 days

The update was successful, but appears to not resolve the issue with a previous update. We will be activating our backup firewall to allow our vendor to further diagnose the issue

05/06/2020 16:57 PM 18 days

We've received an emergency hotfix from our firewall vendors to apply to all edge controllers. We will be applying the hotfix tonight at 11PM Eastern and rebooting the ingress routers. During the install all inbound services will be interrupted until the reboot completes. We anticipate the update taking up to 10 minutes to apply. 

Expired certificate ROCKERDUCK (Resolved) Issue / Performance
1 days

certificate updated at 4:23PM Eastern

04/23/2020 20:19 PM 1 days

The SSL for rockerduck has expired and was not swapped out. The certificate has already been renewed and is being distributed to the front end nodes

Database rollback (Resolved) Issue / Performance
3 days

During a schema upgrade dry run, a failure was detected which caused our monitoring software to rollback the latest backup on disk. This has unfortunately reverted changed from the past 48 hours meaning additions and changes to exchangedefender domains, users etc have rolled back. We are pulling the offsite copy from this morning and are working to restore the correct data structure.


Mobile Email Connectivity (Resolved) Investigating / Notice
12 days

As of 10:07 EST everything is operating normally and there have not been any new reports of an issue.

04/13/2020 12:33 PM 11 days

Mobile email connectivity is reported to be impacted for users on Louie. We are actively investigating the issue.

GLADSTONE offline (Resolved) Outage
18 days

Service was restored around 2:13 PM Eastern

At 1:51 the OS on the primary mailbox server forced a restart. At the time of the reboot, the replicas were 8 minutes behind and could not automatically fail over without force. Upon checking the crashed server, it was already in the process of loading into windows. We elected not to force a fail over as we did not want to lose the pending 8 minutes of data. Prior to the reboot, we had reports from various partners regarding IOS connectivity and OWA connectivity. We're still monitoring the situation, but at the moment all services are reporting operational.

04/06/2020 17:56 PM 18 days

Our monitoring has notified us about access issues to GLADSTONE starting at 1:53 PM Eastern. We are actively investigating the issue.

Mobile Email Connectivity (Resolved) Investigating / Notice
18 days

Mobile email connectivity was impacted for users on Gladstone and our backend team has made an adjustment on the server and expect mailflow to return to normal on mobile devices for these users.

If you have users reporting issues from earlier today and have not yet opened a ticket we request that you please have your users try again as this is reported as corrected and first users are reporting fully restored functionality.

Thank you for your patience and understanding at this time.


Network latency (Resolved) Issue / Performance
23 days

Network access and latency issues were resolved within minutes of this NOC alert being opened. We have resolved the issue temporarily and will put in a more permanent fix in the off peak hours during maintenance cycle.

03/31/2020 15:52 PM 23 days

We are noticing increases in request latency for bot web UI portals and mail flow. We're currently investigating the cause.

Primary firewall firmware update (Resolved) Investigating / Notice
23 days

At 11PM Eastern we will be performing an emergency firmware upgrade to our core ingress firewall. During the upgrade, all services will be temporarily interrupted for five minutes. If we run into any issues after the upgrade, we will activate our backup ingress firewall which is running the current configuration and firmware. 

ExchangeDefender Network Performance (Resolved) Issue / Performance
3 days

We are working on several network services today as we anticipate the growth in usage across all our services. We are currently experiencing degraded performance on our database servers which is slowing down and timing out some services.  More capacity is being onboarded, we appreciate your patience with us while everything syncs up.

Exchange 2016 Performance (Resolved) Issue / Performance
5 days

Service was fully restored around 11AM Eastern with no further issues reported by our partners

03/19/2020 14:15 PM 5 days

We are actively working to resolve intermittent access issues to 2016 mailboxes. Mobile phones seem to be the least impacted, followed by Outlook and then OWA. We anticipate all connections stabilizing by 11AM Eastern but customers should see performance increases as we update throttling policies.

Web UI Latency (Resolved) Issue / Performance
6 days

We've implemented our work around solution and we're seeing dramatic improvements in response time. We will continue to monitor the performance and work on fixing the underlaying issue with mail routing through LAX. At the moment, there are no observed delays

03/11/2020 13:46 PM 13 days

We've reconfigured our LAX data center to proxy as much mail traffic through Dallas to alleviate any mail delays that were observed yesterday as we continue to repair an issue in LAX. Unfortunately the increased load in Dallas is causing occasional timeouts on sites like support.ownwebnow.com, admin.exchangedefender.com, etc. We are close to implementing another workaround solution which should resolve the mail delays from yesterday without impacting the performance in Dallas

Potential mail delays (Resolved) Investigating / Notice
6 days

We've implemented our work around solution and we're seeing dramatic improvements in response time. We will continue to monitor the performance and work on fixing the underlaying issue with mail routing through LAX. At the moment, there are no observed delays

03/10/2020 14:35 PM 14 days

We've re-enabled our LAX datacenter and we are processing mail in both locations. We will monitor the load distribution over the next 15 minutes and update this NOC when service is fully operational

03/10/2020 14:18 PM 14 days

We're working on an issue with our secondary datacenter. For the time being, we've disabled mail from being accepted by our LAX and routing everything through our primary data center. We haven't seen any spikes in delays, but there is potential for 5-10 minute delays. We will update this post when all data centers are back to full operational status.

Incomplete SMTP mail logs (Resolved) Investigating / Notice
29 days

Our LAX cluster suffered a failure when logging SMTP transaction logs for search in the web UI for messages that arrived in the past 24 hours. Users will still be able to search and view the meta data of a message (from, to , subject, timestamp) but will not see the actual delivery logs if the message was processed by our LAX data center.


If you require the logs for a specific message, please open a support request with the meta information of the message and our staff will pull the logs from the backend.

Mailbox failover in Exchange 2016 (Resolved) Outage
29 days

As of 3:51 all users are confirmed as failed over and operations are back to normal. We will continue to work with Microsoft to ensure this bug / issue is properly diagnosed and leads to a more permanent solution. 

02/25/2020 20:38 PM 29 days

Service restart has forced the mailbox instances to start failing over. We should have services fully recovered for all users shortly.

02/25/2020 20:24 PM 29 days

We are experiencing a minor outage affecting 84 mailboxes on our Exchange 2016 platform. Our teams are trying to determine why the failover did not occur correctly, and out of abundance of caution, we aren't terminating the system call.

In the meantime, all 84 users affected by this Exchange database outage can still send and receive email at https://livearchive.exchangedefender.com while the maintenance is underway.

Intermittent outages on 2016 (Resolved) Issue / Performance
20 hrs

All work was completed successfully and things have been moving smoothly since Saturday morning. The new configuration was tested and failover is now behaving as intended. 

02/21/2020 06:50 AM 4 days

We successfully replaced the core router with a spare that we had on hand. We will still replace the unit with the shipped replacement from Cisco Friday or Saturday evening. This NOC will be updated once the swap out is completed.

02/21/2020 05:49 AM 4 days

We will be performing the hardware swap out in 10 minutes. Service is expected to be back online by 1:15AM Eastern

02/21/2020 05:11 AM 4 days

We're in the process of reprogramming one of our spare Cisco routers in order to temporarily replace the current master. We've elected to go this route to allow us to flip back to the original master without a conflict of configurations and addresses. 

We will attempt to activate the spare router in one hour at 1AM Eastern. During the swap, users will not be able to access their mailbox. We've allocated a 45 minute window to perform the swap. If we are unable to complete the swap in the time frame we will revert the changes and continue to work on a temporary solution.

02/20/2020 23:03 PM 4 days

Our team has been working on this issue all day and after consulting with Cisco we are going to attempt a temporary solution that will hopefully minimize the impact to the 2016 infrastructure until the routers can be managed on Saturday, during our weekly maintenance interval where we can minimize impact and continue to serve our clients with minimal interruptions.

We understand this is a serious issue as any impact to performance is an impact to the productivity of the people we serve. We have been fortunate that Outlook is fairly resiliant and have only had a few complaints. Mail isn't bouncing, and disconnections only last from a few moments to a few minutes and happen every few hours. We will attempt a workaround at 2AM EST and hope to be able to restore performance to normal at least until Saturday when full maintenance cycle can be performed. In plain English: we're trying to patch it along so it can make it to the weekend where we can sustain a potential outage of 30 minutes.

02/20/2020 15:33 PM 4 days

We've identified an issue with a core ingress router to our Exchange 2016 network. We will replace the router during our core maintenance cycle this weekend. During the router replacement, we will actively keep connections from opening while we validate the performance and stability of the replacement router. We'll update this posting during the upgrade.

Dallas Routing Issues (Resolved) Outage
12 days

At 8:12AM our Dallas DC firewall infrastructure suffered a catastrophic failure and failover wasn't instant across all services. Failure in one of the firewalls disconnected several private VPN connections that isolate/separate and protect the ExchangeDefender's internal services. We were forced to restart firewalls, which had to reestablish VPN connections, which required all the services to reload configurations simultaneously as routing work was started to move traffic around and away from Dallas. At about 8:24AM all services have been confirmed as green and back in services.

We have double checked every system and everything is back to normal.

Outage lasted approximately 10 minutes.

02/13/2020 13:39 PM 11 days

Private VPN tunnels are back up and all the log, maillog, ips, services are verified to be back in sync.

02/13/2020 13:30 PM 11 days

As of 8:21 the routing and almost all the applications have been restored. We are still working on what caused the problem and how to mitigate it automatically. 

02/13/2020 13:20 PM 11 days

We are working on a massive outage in Dallas. Service outages have been reported, we are working on it and will provide updates every 15 minutes as they become available.

ExchangeDefender inbound mail delay (Resolved) Issue / Performance
18 days

As of 11AM, the messages that were delayed in our Dallas DC have been processed and delivered. All other data centers processed mail without an issue.

02/07/2020 15:51 PM 17 days

As of a few minutes ago all the new mail is going through the load balancers without delay, the mail that was delivered to nodes that were experiencing DNS/latency issues in Dallas is being processed at the moment and will be delivered in a few minutes.

ExchangeDefender database upgrade (Resolved) Investigating / Notice
6 days

We will be upgrading to a new database cluster at 4 PM EST on 2/6/20

During the upgrade all web UI services will be taken offline for 5 minutes. The services affected will be

  • Admin UI
  • Encryption
  • Web file share
  • Live archive

Mail flow will not be interrupted at all during the upgrade

Exchange 2016 clock drift (Resolved) Issue / Performance
25 days

The issue with the clock drift has been addressed and time is displaying correctly on new messages. No mail was lost or delayed during this issue but some messages got a timestamp of up to 5 minutes in the future.

01/29/2020 16:45 PM 25 days

We've been addressing an issue with clock drift on one of our Exchange 2016 nodes. The drift is less than 5 minutes and should be resolved momentarily. 

SPAM targeting Invoicely (Resolved) Investigating / Notice
6 days

We are currently tracking large amounts of messages from recently registered .site domains that are forging their messages to appear as Invoicely. You may see subjects like the ones below, if some are released to your Inbox please delete them:

Past-due payment notification from Invoicely

Past due invoice notification from Invoicely

Overdue invoice notification from Invoicely

Delivery delays and issues with btinternet (Resolved) Investigating / Notice
27 days

BT Internet DNS issue seems to have been addressed. We have no visibility into their network, however, one company has let us know that several items sent to BT did not make it to recipients mailboxes even though the delivery has been confirmed. Unfortunately, free email accounts and ISP email accounts tend to do this from time to time, the only option is to resend.

01/24/2020 17:41 PM 17 hrs

We are currently working with British Telecom to address an issue on their servers related to their implementation of  synchronoss.net software. What our customers are experiencing, intermittently, are delivery delays while sending messages to btinternet and associated properties. The issue was first reported to us around 9AM EST and was quickly addressed to bypass/mask the ExchangeDefender network and route around the problem until BT has a chance to figure out why their DNS and outsourced mail service are having issues with DNS resolution.

Current error produced on their side is:

421 re-prd-rgin-021.btmx-prd.synchronoss.net Service not available - no PTR record for x.x.x.x


As of 10:15 AM this morning this issue has been addressed for our customers so they can send mail to BT, however, this is an issue outside of ExchangeDefender's control and free/ISP mail staff tend not to be very concerned about email delivery issues in general so we do not expect a quick resolution nor can we estimate an ETA. 

Performance of r.xdref.com (Resolved) Issue / Performance
23 hrs

The issue that caused performance impact and timeouts has been addressed overnight on the backend. We have improved the failover and load balancing of our container infrastructure that handles services that were impacted in our phishing.

01/23/2020 18:05 PM 1 days

ExchangeDefender SQL driven properties (admin, r.xdref.com, etc) are moving slower than usual today, we are allocating additional resources to the instances serving these services and expect the performance to return to nominal by 1:20 PM EST. The issue is within the ExchangeDefender cloud and is being addressed by our team right now.

Large scale phishing attack from Russia (Resolved) Investigating / Notice
6 days

We are presently experiencing a massive phishing attack originating in Russia targeting enterprise users from Office365 and beyond. All messages suggest that the invoice/order/bill/payment/transaction has been completed and encourages clients to click on a link that either has a dangerous malware payload or identity theft attempt. This is similar to the Azure and Amazon Web Services (aws) outbreak that has been gaining momentum for the past week.

If you have ExchangeDefender, you don't have anything to worry about - we're blocking their access to our clients and anything that does slip will be picked up by the ExchangeDefender Phishing Firewall which will block traffic to those dangerous sites.  

If your ExchangeDefender service is managed by a third party (MSP) please check with them directly to see if they have blocked ExchangeDefender Phishing Firewall because that action would put you at risk. 

Public folders offline (Resolved) Issue / Performance
16 days

Public folders are now online in the ROCKERDUCK cluster

01/09/2020 13:54 PM 15 days

We are working to restore access to the primary public folder database in the ROCKERDUCK cluster. Currently, users are unable to access public folders through all mediums. We anticipate restoring service within 20 minutes.

Dismounted database ROCKERDUCK (Resolved) Issue / Performance
17 days

We received an alert about a single dismounted database. The original hosting server is indicating performance issues which has prompted us to activate a switchover for the database. We are working to restore service to the users on the affected database.

Live Archive access issue (Resolved) Issue / Performance
23 days

We have resolved issues with the proxy/redirection/caching that affected some LiveArchive logins. The LiveArchive login now inherits the functionality/style of all the other ExchangeDefender/Wrkoo products.

12/31/2019 20:26 PM 24 days

Some users are having issues accessing their Live Archive accounts, we are working on a solution. We are sorry for the inconvenience this may have caused.

SSL certificate issue on scrooge cluster (Resolved) Investigating / Notice
3 days

We are currently investigating an issue with a certificate chain on one of the Exchange CAS nodes in the Scrooge infrastructure. If you've been disconnected your Outlook will attempt to reconnect automatically after a few minutes. 

LOUIE access issue (Resolved) Issue / Performance
11 days

We are seeing failed requests for 13% of connections to LOUIE. We're currently investigating the issue

Office 365 Outbound issue (Resolved) Issue / Performance
20 days

The issue should be fixed.

Please verify. Thank you.

12/04/2019 14:27 PM 20 days

Microsoft is currently making changes to their new servers. It appears the 365 team is using *.prod.outlook.com for newer servers vs the previous rDNS name of *.outbound.protection.outlook.com . We have added the new host name to the allowed list which should fix it.

We will monitor the situation. Thank you!

Malwarebytes false positive (Resolved) Investigating / Notice
20 days

Malwarebytes have confirmed that the issue has been fixed and the site is whitelisted again.


Thank you for your patience.

12/04/2019 14:24 PM 20 days

We are currently receiving notifications of a false positives for our support site.

Please ignore this as it  is a false positive. We are working on getting this removed. Thank you.

POP3/IMAP/Webmail for Webhosted Mail Inaccessible (Resolved) Outage
14 days

We've opened this issue again as there are still issues accessing services. We've temporarily disabled POP and IMAP connections to provide at least a reduced consistent connection

11/22/2019 15:38 PM 2 days

Service has been restored and mail is flowing. The original and replacement motherboard were both defective but remedied by a replacement motherboard that was overnighted to our datacenter.

11/21/2019 16:49 PM 3 days

We received alerts about the mail service included with webhosting being offline. We are beginning our investigation and will add additional notes to this NOC as progress is made.

Fixed XDPF Rewriting for multiple hashtags in URL (Resolved) Investigating / Notice
4 days

On Tuesday, November 19th, 2019 we addressed a bug in the ExchangeDefender Phishing Firewall URL rewriting software that was improperly truncating URLs. Our partners at ReadyCrest Ltd identified the bug by comparing the length of the compressed, rewritten and raw URL and seeing that we truncate link after the first #.Usage of # tends to be fairly explicit in HTML emails and almost exclusively used for anchor links. These links don't actually connect to another site/document, they are meant to scroll the browser to a specific part in the web page. Typically they are used in a table of contents for longer pages, contracts, etc to move the browser to where the actual content is. The syntax is # followed by the name of the section to scroll to - so there is NO functional reason in HTML to ever have more than one # in the URL. That was key in our design and decision to drop anything past the second # - but we've now discovered that some sites may, in fact, put extra (typically reserved) special characters like # in the URL which breaks ExchangeDefender Phishing Firewall rewrites because it removes portions of the link.To our knowledge, this is the first reported link with multiple # in the URL, and going forward multiple # will be allowed in the URL without truncation.This issue has been addressed.

LOUIE access issues (Resolved) Outage
12 days

Access has been restored to users on the dismounted database. We are now investigating why the automatic failover didn't occur for the dismounted database.

11/13/2019 01:12 AM 12 days

A mailbox node in the LOUIE cluster lost one of its uplinks and has caused one database to become dismounted. Prior to losing the uplink, the node held 6 mailbox databases and five successfully failed over to another node. It is currently unclear why the single database did not fail over to a replicated node, but we will investigate it further after we've restored service. We're in the process of replacing the uplink and service should be restored shortly.

Encryption Portal (Resolved) Issue / Performance
12 days

We are currently investigating the Encryption Portal and some users reporting issues with opening emails and seeing the subject from the Encryption Portal.

Please standby as we take a deeper look into this. Thank you!

Incorrect Invoice Subject (Resolved) Investigating / Notice
13 days

This issue has been resolved. Thank you!

10/31/2019 22:18 PM 24 days

The invoices sent to our clients for November came with the subject "Monthly Services (December, 2019)" instead of November, 2019. We've made appropriate adjustments on the backend and everything is labeled correctly as of 6PM EST, October 31, 2019.

Service Portal peformance (Resolved) Issue / Performance
13 days

This issue has been resolved. Thank you!

10/29/2019 15:37 PM 25 days

We are currently investigating issues with portals in regards to our services.

Please stand by. Thank you.

SMTP log issue (Resolved) Issue / Performance
26 days

The issue has been fixed.

All new email will reflect properly in the logs.

Thank you for your patience.

10/28/2019 17:23 PM 26 days

We are currently expecting some issues for SMTP logs for some domains. Logs are showing blank for a few domains. 

Please standby as we fix this. Thank you!

SharePoint Performance (Louie) (Resolved) Issue / Performance
1 days

This issue has been resolved. All operations are back to normal.

Thank you.

10/24/2019 14:05 PM 21 hrs

We are experiencing further issues on this specific server with SharePoint.

We will update once we have a resolution. Please stand by!

Thank you.

Preview infrastructure (Resolved) Issue / Performance
2 days

The issues were identified fixed with the servers.

Thank you for your patience in this matter! 

10/22/2019 15:12 PM 2 days

We are investigating an issue with our message preview infrastructure, possibly affecting a few users on Encryption, Compliance Archiving, and SPAM Quarantines. The issue should be resolved momentarily if you have an issue with a preview loading just hit refresh.

Thank you!

SharePoint Performance (Resolved) Issue / Performance
2 days

The issues were identified fixed with the servers.

Thank you for your patience in this matter! 

10/22/2019 12:26 PM 2 days

We are getting reports of certain domains not being able to log into their SharePoint sites.

We are investigating the issue and will keep you updated.

Thank you for your patience.

Accelerated update to inbound load balancers (Resolved) Issue / Performance
8 days

We've completed an upgrade to our load balancing infrastructure. These servers and switches take on the traffic from the Internet and pass it on to appropriate inbound nodes in the ExchangeDefender cloud for processing. While we have been hoping to roll out the upgrade to the load balancing infrastructure later in November during the long Thanksgiving holiday, we simply needed next-gen stuff immediately. The event that occurred on Wednesday, October 16th, 2019, would have been fully and immediately mitigated by the new load balancers so we are hoping that this upgrade starts performing well for our users immediately. We activated the new load balancer in our primary location (Dallas) at midnight on October 17th and immediately saw a performance gain on the processing speed for inbound messages.

Admin page (Resolved) Issue / Performance
21 days

The issue at hand has been fixed.

Please verify that the page is working properly.

Thank you for your patience.

10/04/2019 13:56 PM 20 days

It has been brought to our attention that there is some user currently being affected by performance issues on https://admin.exchangedefender.com/

We are currently investigating the issue.

Thank you for your patience. Please standby.

SSL refresh for Exchange 2016 (Resolved) Investigating / Notice
7 days

The certs have been refreshed

09/17/2019 14:06 PM 7 days

Tonight at 11PM Eastern we will be restarting the load balancer service for 2016 to use the renewed SSL for the root domains. Access should only be interrupted from 11:00 PM - 11:05 PM Eastern.

SSL xdref.com (Resolved) Issue / Performance
26 days

Thank you for your patience.

The Engineering team has adviced that the issue has been fixed.

Please have all users affected by this and verify. If you are still facing issues please open up a ticket on Support Portal, please.

Thank you.

08/29/2019 11:18 AM 26 days

We are currently experiencing some SSL Certificate issues with xdref.com. We have identified the issue and are currently working to resolve it. 

In the meantime have users Click on Advance and  select Proceed to r.xdref.com

This is so they can still keep going to their links.

We appreciate your patience.

Automatic Provisioning of ExchangeDefender - Reports Investigating / Notice
27 days

This NOC alert is regarding a timely / deadline-driven issue related to billing:

This month we launched the replacement to XD Sync and it has been working really well in terms of identifying email addresses that are sending mail but do not have valid accounts in ExchangeDefender.

You can read more about it here: http://www.exchangedefender.com/blog/2019/08/exchangedefender-account-provisioning-live/

What we are adding at the moment, and will have ready by EOD August 28th, 2019, is a centralized report and visual indicators for Service Provider/CIO personnel to see which accounts have been added so that there is no surprise on the 1st of the month when invoices go out.

Information will be posted on our blog at https://www.exchangedefender.com/blog

Encryption Portal Error 1 (Resolved) Issue / Performance
27 days

Thank you for your patience.

The developers have advised that this issue has been fixed. It was due to a small bug that was easily and quickly repaired.

Please have any users previously affected by this issue try again with reading any messages.

Thank you.

08/27/2019 19:29 PM 27 days

Some users are experiencing issues with the Encryption portal (Error 1).

We are currently investigating. 

Please standby.

ROCKERDUCK Public Folders (Resolved) Outage
2 days

Access has been restored to public folders.

08/22/2019 16:03 PM 2 days

Customers with public folders are unable to access public folder data. We're working on restoring access as quickly as possible

Della peformance (Resolved) Issue / Performance
6 days

Thank you for your patience in this matter.

The issue at hand has been fixed. If you are still experiencing issues please open up a ticket on here.

Thank you.

08/16/2019 16:45 PM 8 days

We are currently facing some issues with the Della Server.

Please have users use LiveArchive while we pinpoint this issue.

Thank you for your patience.

Autodiscover prompt on 2016 (Resolved) Investigating / Notice
6 days

Thank you for your patience in this matter.

The issue at hand has been fixed. If you are still experiencing issues please open up a ticket on here.

Thank you.

08/15/2019 14:05 PM 9 days

Between the hours of 4am - 930am Eastern, Outlook users would have seen a autodiscover prompt for an account not tied to their organization (billing@travispot.com) in error. The autodiscover prompt was a result of us testing public folders for our 2016 environment and that account was one of our demo public folders. Although no users could see or open the public folder, autodiscover detected it as an available mailbox and begun to prompt users about allowing Outlook to try and connect to the public folder.


The public folder was disabled and we will investigate this further into the off hours

Della Issues (Resolved) Issue / Performance
26 days

We have fixed the issue with Della.

The issue was attributed to some settings that were promptly enhanced in order to prevent this issue in the future.

Please verify and thank you for your patience.

07/26/2019 13:33 PM 28 days

We are currently facing some issues with the Della Server.

Please have users use LiveArchive while we pinpoint this issue.

Thank you for your patience.

Admin Portal/XDREF Feature Update. Investigating / Notice
14 days

New features/Options are now available  in the Admin Portal:

- Ability to upload a file (csv) that contains multiple blacklist entries (domain level)

- Phishing logs (sp,domain, user level )

XDREF Feature Update

Added - When whitelisting or blacklisting, XDREF will now be adding the same rules to any alias's you may have. This includes alias's domains as well (if you're whitelisting on the Domain level) 

r.xdref.com Feature Update Investigating / Notice
14 days

Quick update, we continue to work on r.xdref.com as we roll out new functionality, settings, and ease of use. Here is what we did today:


- Currently working on the ability that if you whitelist or blacklist for one alias, it'll trickle down to all aliases and the main account. 

- To add to that, the functionality to whitelist and blacklist items for an alias domain to trickle down to all alias domain's and the Main Domain.

- Rolled out the wild card feature we have on our standard whitelists and blacklists. Meaning you can now do an entry for :

- *.domain.com - to grab all sub-domains for the domain

- domain.com - to whitelist/blacklist the domain itself

We'll continue to update you with new features as we continue to update the service. 

Della peformance (Resolved) Issue / Performance
15 days

The issues with DELLA is fixed.

We have verified that email is flowing properly.

Again, thank you for your patience in this matter.

07/10/2019 11:32 AM 15 days

We are currently seeing some issues with DELLA.

We are looking into the issue and  are working on getting it resolve ASAP.

Please standby and thank you for your patience on this issue.

r.xdref.com Update Investigating / Notice
15 days

Quick update, we continue to work on r.xdref.com as we roll out new functionality, settings, and ease of use. Here is what we did today:

- Reduced log databases, seeded archive log databases

- Rolled out analytics modules for frequent refresh (redirect to help in case they aren't reading the screen)

- Added more tld validations to make sure hackers can't sabotage the firewall by whitelisting all tlds (our liability)

- Exposed additional monitoring and management control panels for support for immediate remidiation

- Added additional warnings for URLs (whitelisted or not) that lead to executable content

ExchangeDefender xdref.com Phishing UI Changes Investigating / Notice
16 days

We have been rolling out new features and fixing bugs / tweaking things on r.xdref.com all day, here is the summary so far:

- We have made the Link: window more prominent

- We have added a link to Report Issue for phishing advice & support purposes.

- We have been optimizing the search and redirect code all day, to automatically review/whitelist sites that are visited often.

- BUGFIX: Whitelist core

- BUGFIX: Email redirection, mailto: automatically redirects

- BUGFIX: Redirect on Whitelist, when client clicks to whitelist and is authenticated, system will automatically redirect

- BUGFIX: Change text from "Link:" to "Click below to proceed to the site:"

ExchangeDefender xdref.com Pishing Firewall Whitelists (Resolved) Issue / Performance
19 days

During the final phase of ExchangeDefender Phishing Firewall deployment, our whitelists briefly stopped working. This means that the users were prompted with the info screen every time they clicked on any link, good or bad, which should not be happening. An emergency patch at 2:30 PM EST was deployed and now whitelist and permitted list code is in effect.

Info: By default, ExchangeDefender Pro clients will only see the warning screen under 2 conditions: 1) Domain they clicked on is categorized as a malicious source of phishing/malware or 2) Domain is not one of the top domains on the Internet and is not on the whitelist (typically, this is where phishing content is generally directed to). For most users, warning/redirection screen should seldom be displayed.

SharePoint errors (Resolved) Issue / Performance
27 days

The issues was identified as the servers were not able to boot up after hardware swap for improved performance.

This was quickly fixed and the performance of the server was enhanced.


Thank you for your patience.

06/28/2019 12:43 PM 26 days

We are getting reports of certain domains not being able to log into their sharepoint sites.

We are investigating the issue and will keep you updated.

Thank you for your patience.

General Internet Outage (Resolved) Investigating / Notice
2 hrs

The outage seems to be reported as fix about 16:00 EST yesterday.

It seems Verizon was responsible for the outage due to a widespread BGP routing leak. This affected a number of Internet services and a portion of their traffic.

BGP acts as the backbone of the Internet, routing traffic through Internet transit providers and then to services. There are more than 700k routes across the Internet. By nature, route leaks are localized and can be caused by error or through malicious intent.

Thank you for your patience in this matter.

06/24/2019 13:52 PM 21 hrs

It has been brought to our attention that several popular sites like Google, Amazon, Reddit, and Spectrum — just to name a few — are experiencing issues this morning. Those issues appear to have begun around 6 or 7 AM ET.


Problems appear to be clearing up as of 8:40 AM, it's likely going to take some time before everything is running smoothly again. Reports are still going up on DownDetector as of writing.


We will continue to monitor the situation as it develops.

Thank you for you patience in this matter.

Admin Maintenance (Resolved) Issue / Performance
4 days

If your users haven't received their scheduled Quarantine Reports, please know that you can resend the reports. 


Simply log into (https://admin.exchangedefender.com) with either your SP or Domain level account.

Once there, select accounts, and find the users who are reporting that they haven't received it. Once they've been selected, click on the Actions button then click on the "Resend SPAM Report" Button. 


If you're still experiencing issues with receiving SPAM Messages, please let us know and we'll be more than happy to take look into it. 


06/20/2019 15:00 PM 4 days

The issue with the admin site (https://admin.exchangedefender.com) and ExchangeDefender Encryption portal (https://encryption.exchangedefender.com) has been addressed. The issue was related to some services exceeding connection counts (similar to "all circuits are busy now") in which some clients were displayed the error when the connection to the backend services could not be made. Out of abundance of caution, we put up a maintenance notice while resource adjustments were done. Everything is back to normal, mail is flowing.

Specifically, the issue was with the database facilities used for event logging. It did not affect the actual mail processing services.

503 Phishing Links (Resolved) Issue / Performance
8 days

The issue has been fixed. It was found out to be an issue with our hosting company.

Thank you for your patience.

06/17/2019 13:26 PM 7 days

We are currently getting error reports about phishing links redirecting to an error 503.

We are currently working on fixing the issue.

Warning Messages (Resolved) Issue / Performance
8 days

As of noon EST, the issue has been addressed and we have not seen another instance of a malware related DDoS. The problem has been addressed around 8:30 EST, fixed, put back into production for testing at 11AM and into full production around 11:30. As of 2PM EST, everything is in the clear.

For the messages that were received previously and that were affected by the DDoS, the malware scan only disarms the dangerous HTML entities. All legitimate email clients send messages in both HTML and text content, and they will show HTML version if available by default (hence why some saw the warning). To see the actual message just right click, select View Source, and you'll see the text version of it.

As of 9AM, this bug affected 117 messages of which 82 came via typical "marketing" junk source from Indeed, Constant Contact, and vresp.

We believe this problem will not show up due to a change in how we're logging and disarming these bug/tracking entities, but we will keep an eye on it as usual.

06/14/2019 13:22 PM 10 days

We are still working on the issue that was causing the problem, and have disabled the scan that was being affected in order to assure email delivery.

We are investigating and will provide an update when the problem has been resolved.

However, all mail should be arriving without an issue and without the DDoS notice.

If the message had critical / important information (in our searches it appears to have overwhelmingly affected mass mailing platforms like indeedmail, constantcontact, vresp) you can still see it by viewing the source of the message.

Just Right click, select View Source and you'll see the text version of the message.

06/14/2019 12:49 PM 10 days


We are investigating a reported issue regarding DDoS warning messages this morning as we've had a higher than usual number of messages crashing the security/antivirus engine during the scan.


We hope to have it resolved shortly.

In the meantime, if you can ask the client to resend the message, please do.

Minor mail delays (Resolved) Issue / Performance
20 days

Mail flow speeds have returned to normal

06/04/2019 16:40 PM 20 days

We've enabled LAX and will be smoothing out the queues between the data centers for the next 15 minutes. Mail flow should resume normal speed by 1PM Eastern

06/04/2019 15:48 PM 20 days

Our monitoring begun alerting about latency and packet loss issues with our LAX data center starting at 11:31AM. To mitigate any potential issues with delivery, we've elected to pause mail flow going to our LAX datacenter. While LAX is paused, Dallas will see mail delays up to 5 minutes but is able to handle the overall load. We will enable LAX once we see the packet loss subside. 

SPAM Issues (Resolved) Issue / Performance
20 days

We have received reports of a large SPAM outbreak going through our Inbound network. 

The messages are of the sextortion variety and are claiming to have sensitive information/video/recordings of users and that they have to pay a Bitcoin ransom so that the perpetrators do not release the footage. 


Please do not fear as the messages are benign in nature. If you or your clients are currently receiving messages like that, please open a support ticket and attach the unforwarded (we need the ORIGINAL) message in the ticket so we can analyze the headers and send you what domains to blacklist. We'll also use the message you send as a sample to track the different variants of the messages so we can create rules and enforce them immediately.


We appreciate your patience in this matter. 

Issues with Della (Resolved) Issue / Performance
20 days

The issues with our DELLA Cluster has been resolved. 


One of the transport hub services wasn't running properly so we had to restart the service and mail began flowing properly. 


If you are still are experiencing issues with this, please let us know via a Support Ticket and we'll be more than happy to assist you. 


06/03/2019 12:07 PM 21 days

We are currently facing issues with Della server.
We are investigating and fixing the issue. 


Thank you for your patience.

Webfile Server Upgrade (Resolved) Investigating / Notice
20 days

Upgrade completed

05/30/2019 20:35 PM 24 days

At 9:00 PM Eastern we will be upgrading the Web File Server service in order to support larger downloads/uploads. The upgrade is planned to take less than 10 minutes to complete. During the upgrade, customers will receive a maintenance notification regarding WFS until the upgrade is completed.

Office power outage (Resolved) Outage
24 days

We have currently regained powered back at our main office.

We will monitor for 24-48 hours to verify that the issue has been fixed.

Thank you for your patience.

05/30/2019 11:37 AM 24 days

We are currently facing power outages at our main office. 


At the moment it's not affecting any mail flow and everything should be running as normal. 


We are monitoring the situation and will update ASAP.

05/29/2019 13:36 PM 25 days

We have currently regained powered back at our main office.

We will monitor for 24-48 hours to verify that the issue has been fixed.

Thank you for your patience.

05/29/2019 12:46 PM 25 days

We are currently facing power outages at our main office.


At the moment it's not affecting any mail flow and everything should be running as normal.


We are monitoring the situation and will update ASAP.

Replacing predicted failure disk (Resolved) Investigating / Notice
12 hrs

Completed without any problems.

05/23/2019 15:39 PM 1 days

On Friday evening between 9PM - 11PM we will be replacing a bad disk in the LOUIE cluster which currently holds public folder data and transport service data. During the maintenance mail flow will be uninterrupted for domains that have "cas.louie.exchangedefender.com" as their delivery point for ExchangeDefender. All public folder access will be disabled during the maintenance period, however, any queued mail for the public folders will remain in queue. Regular mailbox operations will be uninterrupted.

Emergency Maintenance on Inbound nodes 19,41 and 42 (Resolved) Issue / Performance
2 days

After monitoring the network overall during our peak EST and PST hours, everything appears to be back to our normal nominal network performance, and in many cases significantly better than even our ordinary levels considering the elevated mail flow.

05/22/2019 11:20 AM 3 days

ExchangeDefender staff continued to monitor and tune our load balancing infrastructure through our cloud, particularly in our Los Angeles data center. Upgrades, improvements, and other optimizations should significantly increase performance. We will continue to monitor the situation and update this post throughout the day as we have more information but mail delivery should be significantly faster today.

05/21/2019 21:25 PM 3 days

We believe that the issues in Los Angeles have been addressed but we'll continue monitoring these systems. Nodes in this cluster were pulled in and out of production periodically while we tried to diagnose performance related issues (antivirus, DNS, routing, etc).

05/20/2019 16:29 PM 4 days

We've discovered some performance issues in the following nodes:

-Inbound19

-Inbound41

-Inbound42

Our team is currently performing emergency maintenance on those nodes.

If you're experiencing any mail delays and the NDR's are saying they're coming from those nodes, please let us know. Our team is working hard to fix the issues so the delays do not continue., 


If you have any questions or concerns, please put in a support ticket @ https://support.ownwebnow.com or give us a call @877-546-0316


We appreciate your patience in this matter. 

Reverse DNS error (Resolved) Investigating / Notice
9 days

Issue has been resolved if any denials please try again. Thank you for your patience

05/15/2019 15:21 PM 9 days

We are currently looking into errors with Reverse DNS error for some users. We will update as it develops. Thank you.

Landlines (Resolved) Issue / Performance
9 days

Issue has been resolved. Thank you for your patience.

05/14/2019 19:43 PM 10 days

We are currently expecting issues with incoming calls to our landlines. We are working to fix the issue. We will update as it develops.

Level 3 outages (Resolved) Outage
9 days

The outages seem to be resolved. Thank you for your patience.

05/14/2019 19:41 PM 10 days

Currently Nationwide they are reported Level 3 outages. We will monitor the situation as it develops. Please expect slow response/loading times when using our system.

Sharepoint Issues (Resolved) Issue / Performance
8 days

Thank you for your patience. The issue has been resolved.

05/13/2019 15:26 PM 11 days

We are currently experiencing some issues with one of our Sharepoint nodes on Gladstone. 

Our team is currently investigating the problem and will have a resolution shortly. 

If you're on Gladstone and are experiencing issues with accessing Sharepoint, please open up a support ticket and let us know the sharepoint credentials so we can validate connectivity. 

Exchange Defender Quarantine Spam Reports (Resolved) Issue / Performance
15 days

The issue was fixed.

05/09/2019 12:54 PM 15 days

ExchangeDefender has received reports of several users receiving empty ExchangeDefender Quarantine SPAM reports. We are addressing the issue at the moment and will have it resolved today. In the meantime, all users can access their SPAM messages at https://admin.exchangedefender.com or by clicking on the link in the email.

ROCKERDUCK single DB dismounted (Resolved) Issue / Performance
17 days

We noticed an unusual high level of MAPI failed responses for a single database. We begun an active host switch over which should have immediately completed with no service interruption. Unfortunately, the mount didn't immediately finish and is still processing the switch over. This database holds 90 mailboxes. Users on this database will be unable to access their mailbox until the switch over is completed. 

POP3 server mail1 issue (Resolved) Outage
24 days

Service has been restored to mail1

05/01/2019 02:19 AM 24 days

We are investigating an outage on mail1.ownwebnow.com (mail1.exchangedefender.com). Currently all services are inaccessible. This affects some web hosting clients who use the POP services

Pausing mail flow in Dallas 9PM (Resolved) Investigating / Notice
24 days

This was completed without any issue

04/30/2019 17:11 PM 24 days

We will be restarting some Dallas ExchangeDefender inbound node core components at 9:45PM until 10:00 PM on 4/30/19. We will pause mail flow going to Dallas at 9:30PM to prevent any mail flow issues from occurring. Mail will continue to route through our other data centers and there will be no interruption in mail flow.

SSL Certificate Issues (Resolved) Issue / Performance
24 days

We are currently experiencing some SSL Certificate issues on our GLADSTONE Cluster. We have identified the issue and are currently working to resolve it. We appreciate your patience.


Inbound Phone Issues. (Resolved) Investigating / Notice
24 days

We are currently experiencing issues with our inbound phone system. If you require immediate assistance or a callback, please open a support ticket @https://support.ownwebnow.com and we'll be more than happy to assist you. 

We appreciate your patience in this mattter. 

Issues with Public folders on LOUIE (Resolved) Issue / Performance
20 hrs

The public folder issues on our LOUIE Cluster have been resolved. 

If you are still experiencing issues with accessing public folders, please let us know. 

We appreciate your patience in this matter. 

04/24/2019 14:57 PM 20 hrs

We are currently having issues with accessing Public Folders on our LOUIE Cluster. Our team is currently investigating these issues ans will provide an update as soon as it's been resolved. We appreciate your patience.

SSL Certificate Issues (Resolved) Issue / Performance
1 days

We have rectified the SSL issue on our Rockeduck cluster, we appreciate your patience in the matter. If you're still encountering issues, please call us 877-546-0316 or put in a support ticket https://support.ownwebnow.com. Thank you again for your patience.

04/23/2019 15:17 PM 1 days

We are currently experiencing some SSL Certificate issues on our ROCKERDUCK Cluster. We have identified the issue and are currently working to resolve it. We appreciate your patience.

Mail Delays (Resolved) Investigating / Notice
7 days

The issue with the Mail Delays across both the Dallas and Los Angeles Data Centers has been identified and repaired. Mail should start flowing normally within a few minutes. If you have any questions or concerns, please contact our support team via Ticket or call us at 877-546-0316. We appreciate your patience.

Exchange 2016 Database Distribution Rebalance (Resolved) Investigating / Notice
8 days

We have scheduled a rebalance of active hosted database distribution on our Exchange 2016 Nodes. This will have no impact on customer accessibility. 

LOUIEMBOX7 Performance Issues (Resolved) Issue / Performance
11 days

Reboots have been scheduled for LOUIEMBOX7 due to Memory issues. This is scheduled for 930PM EST today. This will have no impact on customer accessibility.

Comcast Blacklist Rejections (Resolved) Investigating / Notice
12 days

We have received two (2) client complaints about email being rejected by Comcast. 

At the moment we aren't seeing issues with delivery to Comcast, none of our IP addresses on any outbound networks are on an RBL (mxtoolbox.com), and manual telnet SMTP diagnostics tests are so far showing no issues either. This leads us to believe that it's a typical Comcast issue (rejecting mail randomly from different servers as they get swamped and overloaded during peak hours). Best practices for sending mail to Comcast or Yahoo is to just resend the message and hope a different server in their infrastructure accepts the message. When there are RBL issues, we immediately open a support request and start working with their postmaster staff.

Email delivery delays on Gladstone (Resolved) Issue / Performance
12 days

The issue was resolved and clients have reported no issues since.

04/11/2019 16:45 PM 13 days

Maintenance task has been completed and we have not seen nor have we received reports of any issues.

04/10/2019 14:00 PM 14 days

Some users on the ExchangeDefender Exchange cluster GLADSTONE experienced significant email delivery delays throughout the day as we rebalanced our storage. This is a one-time event that was a side effect of a major maintenance task required to facilitate a change in processing of our Compliance Archiving mail. Traditionally, we stored ExchangeDefender mailboxes and journal mailboxes on same volumes which during peak hours could create a performance issue, and this maintenance task assured that going forward Exchange bottlenecks will not be a source of issues for clients that remain on our legacy Exchange infrastructure.

If this issue affected your clients adversely, please schedule a migration of their mailboxes to Exchange 2016 with our team immediately and we will make sure that they are moved to the front of the queue.

ExchangeDefender LiveArchive still received and sent messages in realtime allowing our clients to not feel a major performance impact.