p0pr0ck5 / lua-resty-waf Goto Github PK

Functionality was already implemented by calio

• pers. key value
• description
• pattern (could get ugly and expensive real quick- maybe a parse_pattern option?)

can you tell me what repository you use to generate libac.so, because we use Mac OS

It might be possible to get a greater execution speed if the control flow was pre-calculated.

What this would likely involve is precalculating a jump-offset for all rules, for non chained rules this would be one. In normal cases the pointer moves forward by one, either in a chain or when matching in a chain. If however the rule did not match the pointer moves forward by the rule's precalculated jump. That is it does to the end of the chain.

Saves a fair few executions of _process_rule

This should be relatively easy to implement:

• Store in a shm (needs to be user-defined in http{} block)
• deprecatevar/expirevar implemented via timer

This will be useful to provide an abstraction layer to implement more features like bf protection, tarpitting, etc

Rather than rely of the default provided rulesets, allow users to specify a table of modules, and an optional path to search.

Currently FreeWAF will skip over request body data that is larger than client_buffer_body_size. A workaround to this is to set client_buffer_body_size equal to client_max_body_size, but this could lead to severe performance degradation. A better alternative might be to integrate a file scanner, and provide an option to enable/disable file upload checking.

RT

42043 appears twice, first one should be 42042

local function _regex_match(self, subject, pattern, opts)
local opts = self._pcre_flags

since opts will be override by self._pcre_flags, why should pass opts param in function

pattern = "Referer" should be pattern = "referer"

Lookup table is getting a little clumsy, set the default behavior to just set the instance value to the option value, and make other behaviors do the needful.

• transforms
• event logs (severity, target)
• persistent storage

This makes debugging ruleset logic tricky.

When debugging I have noticed an extreme amount of debug output. I propose reducing some of the mostly superfluous output. I propose starting with utility methods. This has an added benifit of simplifying the signatures for these outling methods.

Some possible targets include the operators.*:

_log(self, "Needle is a table, so recursing!")

and

_log(self, "Comparing (greater) " .. tostring(a) .. " and " .. tostring(b))

Some information from this could be included in the log output of process rule instead, possibly grouped with another message.

This has the added benefit of making the utility functions more generic and suitable for use from module context (not just wf context) where logging is not available.

Some work in this area is in my fork.

L:663

Relates to issue #4.

Hi Robert,

Sometimes I get segfault messages from Nginx, if FreeWAF is enabled. Related lines from dmesg:

[ 2983.164425] nginx[4298]: segfault at 414cb62a ip 00007ff720df7a5a sp 00007fff7cb96820 error 4
[ 3043.214933] nginx[4126]: segfault at 41a61e22 ip 00007ff720e07a5a sp 00007fff7cb96890 error 4
[ 3103.362729] nginx[4301]: segfault at 424f33da ip 00007ff720e07a5a sp 00007fff7cb96820 error 4

This is a Debian Jessie box with a default kernel from repo:
3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt4-3 (2015-02-03) x86_64

nginx-extras and libluajit-5.1-2 packages from Debian repo:
nginx -V
nginx version: nginx/1.6.2
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt=-Wl,-z,relro --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_flv_module --with-http_geoip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_mp4_module --with-http_perl_module --with-http_random_index_module --with-http_secure_link_module --with-http_spdy_module --with-http_sub_module --with-http_xslt_module --with-mail --with-mail_ssl_module --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/headers-more-nginx-module --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-auth-pam --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-cache-purge --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-dav-ext-module --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-development-kit --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-echo --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/ngx-fancyindex --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-http-push --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-lua --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-upload-progress --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/nginx-upstream-fair --add-module=/tmp/buildd/nginx-1.6.2/debian/modules/ngx_http_substitutions_filter_module

Backtrace of the core dump:
http://www.emrah.com/files/freewafbt1.txt
http://www.emrah.com/files/freewafbt2.txt
http://www.emrah.com/files/freewafbt3.txt

I have Nginx's debug log too but it's really big and I don't know how to find the related lines.

regards
emrah

Defaulting defining 'oij' is bad, make it the default but wrap it in an option

Missing a self param in the recursive call

modsecurity has some rules like "prefix", although this operator can change to PM to satisfied your operator, but I think it is not necessary, I think freeway should offer new operator to support "prefix"

process transactions in roughly 300-500 microseconds per request; this equals the performance advertised by Cloudflare's WAF.

But as far as I know, CF's transaction time is 300-500us, less than 1 ms, why equals ?

e.g. base64 decode, etc. should be baked in to _parse_collection and memoized

Do you know "t:none,t:urlDecodeUni" mean the variables do operator twice? one for none transform function, another for urlDecodeUni transform function?

and I get to know you are doing the work that translate owasp's core rule to your own rule set, what's your schedule ？

See how more more (if any) new delay has been introduced by changes like ignore_ruleset (which is inefficient and now runs with every request)

I upgraded to the current development branch and FreeWAF stops to log.

Server1:
Common FreeWAF config in /etc/nginx/conf.d/freewaf.conf:
access_by_lua '
FreeWAF = require "FreeWAF.fw"
local fw = FreeWAF:new()
fw:set_option("mode", "ACTIVE")
fw:set_option("disable_pcre_optimization", true)
fw:set_option("score_threshold", 10)

    fw:set_option("event_log_target", "file")
    fw:set_option("event_log_target_path", "/var/log/nginx/freewaf.log")
    fw:set_option("event_log_buffer_size", 8192)
    fw:set_option("event_log_verbosity", 3)

    -- fw:set_option("whitelist", "127.0.0.1")
    -- fw:set_option("ignore_rule", 41014)
    -- fw:set_option("ignore_rule", 41015)

    fw:exec()

';

The log file permission is OK. But there is no log.

Server2:
There are two domains and each one has their own config in server { } block. "access_by_lua" blocks are same except the filename in "event_log_target_path".

log is OK for one domain but it doesn't work for other. But if I comment the log options for the first domain then the second starts to log.

why 42000.lua 's regex expression use "" in pattern, but others use [=[ ]=].

Hi Robert,

mp4 is not in "whitelisting extensions - media", rule id: 10007

/test.mp3?a=a' union select --> pass
/test.flv?a=a' union select --> pass
/test.mp4?a=a' union select --> forbidden

rule1 -> rule2 -> rule3

when rule1 match rule, ctx.chained = true, rule2 not match rule, but I think you don't set ctx.chained = false again, so when rule3 match again, it also take the rule3' action.

Per the ngx.timer.at docs:

Because timer callbacks run in the background and their running time will not add to any client request's response time, they can easily accumulate in the server and exhaust system resources due to either Lua programming mistakes or just too much client traffic. To prevent extreme consequences like crashing the Nginx server, there are built-in limitations on both the number of "pending timers" and the number of "running timers" in an Nginx worker process.

Since expire time granularity is tracked in seconds, add in logic to only create one new timer per second

Need flexible logging destinations beyond error.log.

• resty socket logger
• plaintext file logger
• logger settings can use set_option

Right now config options are set globally in the init phase. Need to find a way to deliver config options (whitelist, rulesets, etc) on a per-server basis.

1 Is the spend time you test FreeWAF based on Logging mode or not ? that is to say , whether you turn on the debug logging switch?

2 I find that if I turn on the debug logging mode, in my machine, it takes 9-10milliseconds, but 900-1000 microseconds when it turn off.

3 On one request, you have make some cache between rules to reduce exec time. Can we make some cache between requests to reduce response time?

Look into how we can integrate lua-resty-tarpit's state tracking, and investigate creating a TARPIT action

Now that we have persistent storage, we can start tracking bad behavior of clients

• Number of 400 < x < 500 status responses (vs host avg)
• Number of requests resulting in long upstream response time (vs host avg)
• Repeated requests to the same resource
• Request rate (vs host avg)

Will need header filter phase handling for this

Need a new branch for refactoring.

41015

Might be useful for long-term behavior tracking (e.g. averages), deprecating vars instead of expiring, etc

Hey,

This project looks quite interesting. Its obvious you have put quite a lot of hard work in already 👍
It is also pretty remarkable the features available given the size of the code. Working on understanding it all currently.

1. Is there any documentation on which CRS rules are implemented? It might be a good idea to insert a comment above converted rules with the original rule to make it easy to see gaps. I am guessing you might be missing some of the features needed to do the complete ruleset (that and honestly some of the rules would be quite a performance drain!)
2. Re the benchmarks conducted: was OpenResty compiled with lua or luajit?
3. Have you considered using a nginx pre-processor for debug logs? Its fairly common for large systems (e.g nginx) to be compiled with debug logging for greater performance. The pre-processing could leave the debug log call in lua comments so that it runs in production mode without processing.
4. If you happen to be against to this, perhaps consider making debug logging a global property to the worker, then if it is disabled - overwrite the _log function with an empty function. It would be better than nothing. Still, personally I dislike that complexity and would prefer point 3.
5. I would do the same with disabling pcre optimization, its both a user support risk and a mostly pointless option outside of debugging.
6. In LUA string concatenation is doubly expensive as the hashing is done at this point too, when concatenating a few or more strings `table.concat`` is often faster
7. Is there any lists of tests you have performed? I am interested in doing testing as to the performance of various scenarios - to make sure there are no resource exhaustion / DoS attacks lying in wait.

I will continue reading, but I am interested in getting this up and running - and of course contributing back. Feel free to ignore any questions that seem a bit ignorant at all, I haven't got it running yet so I might be mistaken on a point or two.

At a glance this does most of what I need it to do already. And hopefully it will be able to meet our performance requirements (looks to be very very close already).

Thanks in advance for reading all this :)

We needz it

40022/40023

I suggest you can put "transform = xxx" from rule.opts to rule.var.opts, because var will be token action on transformation before match.

replaceComments, removeWhitespace, compressWhiteSpace, urlDecodeUni

request_headers.host => opts = { key = "specific", value = "host"}

how to handle request_headers.host and request_headers.user-agent ?

to be opts = {key = "specific", value = {"host", "user-agent"}} ?

Hi Robert,

FreeWAF prevents to access the pages when its uri contains "rsa" or "dsa"

www.mysite.com/xxxrsaxxx/ <- forbidden
www.mysite.com/xxxdsaxxx/ <- forbidden
www.mysite.com/xxxyyyxxx/ <- pass

Rule id: 90002

Currently hard coded to ngx.INFO

Can I ask the reason for an explicit whitelist / blacklist? From my understanding it would function the same as using allow/deny from the nginx access module which are always executed before LUA.

The main application I have brainstormed is when integrating into a larger application (like we are looking to do). Such an application is likely rare, and in most cases such a system would want to do this outside of the WAF layer for additional control.

Thanks

Right now everything runs in the access phase. If we want to pursue behavioral analysis, we'd need to look at capturing more data like response status, times, and header/body content. A few problems with this:

• Need to figure out an efficient way to instruct exec() (or build a new model) to only handle rules of a single phase- having every phase handler run through -every- rule (and skip over phase mismatches) seems like a big waste, but need to profile that if it proves to be an easy option
• Users would need to configure multiple phases manually. This seems like it might be too much responsibility to put on the user
• How would we handle chaining?
• Do we use ngx.ctx? A shm? Re-run access phase rules in filter phases (eww)?

Hi Robert,

I installed the current master branch to Debian Jessie box with Nginx 1.6.2 from Debian repo.
I get "module 'resty.upload' not found" error message.

Is it possible to put this module under FreeWAF's directory tree like "inc.resty.cookie"?

regards