Opened 11 months ago

Last modified 9 months ago

#2134 infoneeded_new enhancement

Handle ddos attacks better on core services

Reported by: meeh Owned by: meeh
Priority: maintenance Milestone: soon
Component: trac Version: 0.9.32
Keywords: Cc:
Parent Tickets:

Description

Want to bring up a discussion about ddos protection, specially on the "internet endpoints" of our core services, like trac, main page and so on.

I've been looking at https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html for the trac service, but unsure if it's enough since it seems it's apache itself that's failing.

Has anyone experience with mod_evasive ( https://github.com/shivaas/mod_evasive ) or similar software?

kytv in his time put up an fail2ban service, and I've updated it's config now due to configuration changes in other software on the server. Let's hope that's help.

How should we handle ddos in general? Should we use a service, put more resources on boxes, or any suggestions?

Subtickets (add)

Change History (2)

comment:1 Changed 10 months ago by aargh

  • Status changed from new to infoneeded_new

It depends on what you're getting hit with, slowloris, SYN floods, aggressive/bad behaving crawlers, GFW backscatter, and/or POST floods by botnets trying to send out spam & register on the site?

Most of these problems can be solved with ModSecurity? & IP persistence storage paired with HAProxy (stick tables) or even basic iptables (hitcount) rules to rate limit requests before they even hit Apache.

Stubborn botnets that are 'stuck' on a site can usually be stopped with a mod_rewrite (or modsecurity if a counter is needed) rule to deny the combination of user-agent, referrer headers, and cookies that are present or missing. BPF filtering could be another attractive mitigation.

Comodo offers free rules that extend OWASP and are trivial to modify to your needs. But if you go down that route, be sure to run modsec-sdbm-util periodically to purge the db otherwise ip.pag will probably grow unbounded and cause Apache to become unresponsive under load.

If you have time to roll something custom like a 'self hosted' Cloudflare, consider looking into and experimenting with lua-resty-waf or Tempesta FW. If a third party service would be considered acceptable, try reaching out to deflect.ca by eQualit.ie (if you are *very* adventurous you could self host their mitigation software based on Apache Traffic Server but that would require a lot of upkeep).

If you stick with fail2ban, be sure to remember to periodically clear out it's sqlite database and run a recent release. Also, (more common on version 2.2) Apache might need to be restarted on a regular schedule to prevent memory leaks -- even in production.

That's about all of the suggestions I can offer without reviewing sanitized logs, ss/netstat outputs, and actually diving in during an active disruptive event. HTH.

comment:2 Changed 9 months ago by slumlord

I have set a basic rule in pf that blocks 99% of traffic from hosts that display excessive usage for my public facing services (static files, no dynamic content at all). I agree with aargh, the type of excessive/malicious traffic would determine the manner and level of response. Thanks for the great post, aargh.

Note: See TracTickets for help on using tickets.