Cloud Website Firewall – Sucuri StackPath CloudFlare
If you are in the business of hosting, or developing websites, then we highly recommend a Website Firewall or Web Application Firewall (WAF). Attacks against public facing websites are non-stop, and relentlessly taking away from resources that could be used for legitimate visitors. If you watch these things, the number of bots, scrapers, attackers, and foreign countries hitting the logs is likely high, and it’s always happening, even while you sleep. Website Firewalls used to be primarily appliance only (Imperva, F5 WAF), and some can be virtual machines (Barracuda), but this isn’t very cheap. The overall strengths are that these cloud tools are very easy to implement (DNS), and you can quickly cut back to the original provider if issues arise. Website Firewalls can be used with shared hosting or dedicated hosting.
We have been using Cloudflare, Incapsula, Sucuri, and StackPath as a test to handle websites with around 10k visitors a month, and 100-150k pageviews. During these tests, we’ve identified some key weaknesses, and in some cases major issues with these Firewalls based on how you deploy them. I’m more of a ‘hands-on’ ‘real world’ kind of Engineer so I will save you from all of the marketing and feature breakdowns. Let me tell you what works, worked, and wouldn’t work at all so you can avoid the headaches.
What are the risks to websites?
Web servers are typically not monitored 24x7x365 for attacks, login guessing, and other attempts to compromise them. It can be costly to have a team responding with this kind of availability, and constantly checking for issues that aren’t related to monitoring for the site going down, or other performance issues is time-consuming (i.e. Examining the logs for attacks, manually blocking IP addresses). The threat landscape is always changing and evolving, requiring constant changes to defenses. If a server gets hacked in the middle of the night, does anybody know about it? You do if it goes offline, but not if the code is injected into the site.
Updates for PHP and other packages on the server are not as frequent as Drupal. Other examples, SSH, MySQL, Curl, ImageMagick, Apache, and web related plug-ins that can be attacked from the outside, without logging into the website. These packages have a long history with hundreds of vulnerabilities over the years, especially PHP. Small bugs are strung together to make big hacks, and there are dozens of these bugs in those packages, some fixes introduce new bugs to patch old ones. Upon manual scan, a hacker will make those connections by researching the versions.
Old Software/Plug-in/Platform versions running on web servers are numerous and need constant maintenance to keep them updated. Even with the most current versions, unknown bugs (to the vendor) exist which are leveraged by attackers before the vendors know or implement a fix. For some of these risks, there is no fix that is possible to apply until the vendor addresses it. This can leave months of vulnerability if say a Drupal attack is known to hackers, but not yet available for a fix.
Attacks that flood the website with large amounts of data (aka Denial of Service) are not protected by the web server, or by Linode’s network. We currently would have limited ability to block a flood attack or one from multiple sources from impacting the website. Attackers have in the past done this in the past and requested a ransom from the website owner to stop the attack, and we’ve seen them last for days, even weeks. These are common and have been going on for 15+ years, with attackers constantly adapting to evade new protections. There will always be a way to flood systems and crash them, either by volume or bugs. It’s hard to beat when they target you. We (as penetration testers) could take down many websites with a single
Malware/Viruses can be implanted on the website without our knowledge. This can occur without the attacker having full control of the site, and by exploiting a bug which lets them upload and link a file which is run each time a user visits the website. The user does not have to accept, permit, or see any messages to become infected. We’re scanning for malware now, but not preventing it from being dropped. It’s better to block the attacker from ever uploading it, especially because he may figure out a way to make it undetectable by changing it rapidly.
Fraudulent website files can be uploaded to ‘Phish’ users of the public with spam campaigns. i.e. www.yoursite.com/Office365.html is injected via a bug and spammed to the internet to steal credentials. People click those links and visit our site as part of this fraudulent scheme. This happens to 10s of thousands of sites daily.
Site reputation issues are a backlash caused by Malware/Virus infections. Users who
leverage any one of dozens of products that provide virus, web filtering, or safe browsing features can be blocked from visiting the website after an incident. Petitioning each organization to remove the site can take hours, weeks, or longer, as there is no guarantee that many of these free services will prioritize, or honor the request, even after it’s remediated. (64+ vendors, i.e. Chrome has found this site to be malicious, and has blocked your attempt to visit). Users have no workaround to reach the site in many cases without the involvement of IT to the whitelist.
How do the vendors address these issues?
*** Note 7/4/2018 – Stackpath has launched a new CDN/WAF front-end for future users. Is it better looking? Yes. But be aware, I tried to make an account and migrate my site, and it was fraught with errors. I couldn’t create a CDN instance without doing a support chat, and it disconnected so I had to queue all over again. The rep didn’t understand what I meant by ‘old site’, even though they had just done a global announcement of a ‘new’ Stackpath. I discovered that creating a WAF alone (something they do offer for $10/mo) wouldn’t let me adjust the SSL certificate properly, so my site was broken. Actually, my site was broken the entire time I tried to do the move. I deleted from the old site in desperation and lost many of my settings. I admit it wasn’t something I planned, but how hard could it be? Why do I get ‘Internal Error’ when I’m simply adding a site? Strange, and signs of possible bugs to come. The last issue is they are now charging $1/mo for WAF rules, which in the previous version was Firewall rules. This includes things like whitelisting IPs, countries or blacklisting specific hosts. In my case, I like to proactively whitelist the IP blocks of certain organizations. But wait $1? At that rate, I’d be paying like $200 more a month on top of my $10 plan. Am I being forced into that later this year? This blog needs work, and I don’t know what Stackpath is thinking but I may on the hunt for a new WAF. Stackpath may be turning into a nightmare with a new front-end, and support you’d expect for the money. TBH I’ve had so many bugs many of the WAF features are disabled. Why? Because they won’t address the root-causes of the bugs. I left Sucuri because iPad couldn’t download a simple PDF, a problem they said Engineering was hard pressed to fix, and still isn’t resolved nearly 9 months later. What’s wrong with these providers nowadays?
StackPath a comprehensive choice, and well structured GUI overall. The WAF had all of the configuration ability we needed and is a steal at $20/mo for 5 sites @ 200GB transfer. Website load times are fast, and the CDN works well. We had to tweak some security features, but overall found a robust choice of options. This vendor resolved many issues we had (listed below) with Sucuri and Cloudflare. In my opinion, this is the most superior cloud WAF I’ve ever used, given that it took only 15 minutes or so to figure out and setup. From there, we were able to do live troubleshooting on any blocked traffic. We highly recommend them for WAF, and CDN services as a strong competitor with a great website design.
Real-time log viewer has a ton of information on the blocks and passed traffic
Configuration options on the Firewall are great, and offer plenty of tweaks to avoid false positives
Caching has advanced configurations, didn’t cause any issues in our testing
Great graphs, and real-time monitoring on the dashboard are excellent
Great set of menu options, and reports can even show you what edge sites were hit (source countries)
We are unable to recommend the WAF for any high traffic, or advanced (custom) websites, as there were issues in production with this Firewall, listed below. Overall,
UI is very basic v1.0, lacks organization and robustness of a full-featured product.
Support is e-mail only, no way to call, or chat. Think of the SLA as a response of “We have assigned this to our Engineers”. The average response was 4-48 hours, and always very basic, like saying to whitelist things or change a setting. Often missing key details provided in the ticket, and not addressing all of the questions posed. We opened over a dozen tickets, mostly for issues with a clients site. Folks I spoke with were in Brazil.
Can’t upload attachments, or screenshots to support tickets, have to link them in some way. Very little information gathered at this point, the form is basic doesn’t ask for much, and you don’t get much in return.
Support can’t see anything that’s not in your own console. They have no reporting capabilities if something rolls off the real-time window. Have a user that was blocked 2 hours ago? Tough luck, there is no way to see the event once it’s off the console. You have to wait for the 1 time per day update of a window called ‘Audit Logs’, which is actually an attempt at making some kind of report. The live monitoring here is a nightmare, and if you have a busy site, you will be underwater trying to find logs, because they disappear so quickly. The real-time log viewer only has 50 lines and is NOT searchable in any way (except CTL-F). I spoke to support, and they said that ‘we want searching too, use API Splunk instead’. Wait, why do I need another client to make up for what’s lacking on this site?
No way of generating reports on-demand, just a simple report by e-mail. Can’t export the reports on the website, must wait for the report e-mail. You have to print the ‘report’ page to PDF. Repeat, you can NOT generate a downloadable report or e-mail one on-demand. You have to wait for the daily, or weekly e-mails. There is absolutely nothing you’d want to deliver to a client on here.
Users being blocked that don’t show up in the Real-Time window, roll off the log. Can’t get that information in the Audit Logs until it updates once a day. Forced to use the API to import into Splunk, otherwise, you can NOT troubleshoot issues on a high volume website.
Users being blocked with ‘Unknown ID’ for a block type
No way to bypass Firewall in the console, only a developer mode to non-cache certain IPs. Can’t turn it off, you have to switch the A record back. If you moved DNS over to Sucuri, you are screwed. There is absolutely no way to turn it off and pass the traffic thru, like Cloudflare. We had to change our NS back to DynDNS to get around a problem, and it took forever to finally switch.
The block page looks like an advertisement and does not support customization. It tells the person being blocked how Sucuri is protecting them from threats and provides a bunch of technical information. Often, the reason for blocking was not specified (Code Unknown), or it was in specific (DDoS Attempt Blocked). If you e-mail support, they can’t ‘reveal the logic of the check’, just whitelist everything instead. It blocked somebody from downloading a file out of /system/files/file.pdf, but we don’t know why.
Caching and Optimization:
Sucuri recommended full caching, and it was a nightmare for our dynamic site. We had to roll back the setting to respect the origin sites ‘Https Headers’, otherwise all kinds of things were breaking.
Cloudflare is great for the CDN side of things and has many options. We’ve been using it since it first came around, and I think it’s grown tremendously in the value it can offer. Unfortunately, the WAF is not as configurable as we’d like, and so for this platform, it may be a good choice but should be reviewed carefully against requirements. The #1 takeaway from CloudFlare, in my opinion, is the amount of malicious activity they have on the network. I have personally investigated dozens of phishing, malware c2c, and attacks where CloudFlare IPs were involved. Requests to take down the content were not successful. There are reports about this on the internet abroad, and I think that has an impact on where I’d prefer to host my data. I have not confirmed that CloudFlare has a low reputation for its IPs in any place, but I do think there’s a likelihood that using them could pose a risk to blocking from large enterprises. Check any one of the IPs on Virustotal as a search, or abuseipdb.com and it looks similar to a TOR node. Also, there are tools like CloudPiercer that attempt to determine the real hosting IP for servers behind CF, a concern where it used to create direct.domain.com by default, to let you get access to the host. Overall, the configuration ability of the WAF and lacking options like custom rules, or Geo-Blocking won’t work for us.
Incapsula was not tested in production, and mostly we found it to be expensive with regard to securing small websites for SMBs, or even a medium business. You could not get all of the features with the other provides above without paying $70/site per-month for the pro plan, or more for business. I felt that the web UI was OK, but configuration ability, and overall value was lower than StackPath when it came to CDN. We’ll add more if we go back and review them in production.