Load_secrules doesn't effect. about lua-resty-waf HOT 13 CLOSED

-- translate and add a SecRule files to ruleset defs
function _M.load_secrules(ruleset, opts, err_tab)
ngx.log(ngx.ERROR, "===========================load_secrules function!!!=======================================\n\n")
local rules_tab = {}
local rules_cnt = 0
local f = assert(io.open(ruleset, 'r'))
while true do
local line = f:read("*line")

            if line == nil then break end

            rules_cnt = rules_cnt + 1
            rules_tab[rules_cnt] = line
    end

    f:close()

    local chains, errs = translate.translate(rules_tab, opts)

    if errs then
            for i = 1, #errs do
                    if type(err_tab) ~= 'table' then
                            ngx.log(ngx.WARN, errs[i].err)
                            ngx.log(ngx.WARN, table.concat(errs[i].orig, "\n") .. "\n\n")
                    else
                            table_insert(err_tab, errs[i])
                    end
            end
    end

    local name = string.gsub(ruleset, "(.*/)(.*)", "%2")

    _calculate_offset(chains)

    _ruleset_defs[name] = chains
    _ruleset_def_cnt = _ruleset_def_cnt + 1

end

I changed the code. Add ngx.log, but I've never seen it in trhe error log.

The rule(add by load_secrules) doesn't work. Can you please help me on this?

from lua-resty-waf.

Important error log：

2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil
2019/03/28 21:24:20 [error] 4860#0: [lua] init.lua:14: nil

2019/03/28 21:24:36 [info] 4896#0: *3 [lua] waf.lua:405: _merge_rulesets(): [c1f1953db6528b86c689] Adding ruleset REQUEST-901-INITIALIZATION.conf, client: 112.65.119.119, server: cert.placuna.cn, request: "GET / HTTP/1.1", host: "cert.placuna.cn:8080"
2019/03/28 21:24:36 [info] 4896#0: *3 [lua] waf.lua:405: _merge_rulesets(): [c1f1953db6528b86c689] Adding ruleset REQUEST-913-SCANNER-DETECTION.conf, client: 112.65.119.119, server: cert.placuna.cn, request: "GET / HTTP/1.1", host: "cert.placuna.cn:8080"

The waf:set_option() executed,and print log .

But I can;t get load_secrules log.

from lua-resty-waf.

What is the version of openresty are you using?

from lua-resty-waf.

What is the version of openresty are you using?
1.13.6.2

from lua-resty-waf.

@GumpSun can you please let me know how did you resolved this
Even i am getting init.lua nil error when adding more than one rule.

from lua-resty-waf.

@GumpSun can you please let me know how did you resolved this
Even i am getting init.lua nil error when adding more than one rule.

    init_by_lua_file /home/waf/openresty/waf_lua/init.lua;
    init_worker_by_lua_file /home/waf/openresty/waf_lua/iworker.lua;
    access_by_lua_file /home/waf/openresty/waf_lua/runtime.lua;

    use lua-resty-waf in init.lua like this:
    lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-901-INITIALIZATION.conf")
    --lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-913-SCANNER-DETECTION.conf")
    --lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf")
    --lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-921-PROTOCOL-ATTACK.conf")

lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-931-APPLICATION-ATTACK-RFI.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-933-APPLICATION-ATTACK-PHP.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-944-APPLICATION-ATTACK-JAVA.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/REQUEST-949-BLOCKING-EVALUATION.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/RESPONSE-950-DATA-LEAKAGES.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/RESPONSE-951-DATA-LEAKAGES-SQL.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/RESPONSE-952-DATA-LEAKAGES-JAVA.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/RESPONSE-953-DATA-LEAKAGES-PHP.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/RESPONSE-954-DATA-LEAKAGES-IIS.conf")
lua_resty_waf.load_secrules("/home/kswaf/openresty20190423/waf_lua_ng/rules/RESPONSE-959-BLOCKING-EVALUATION.conf")

runtime.lua file:
local lua_resty_waf = require "resty.waf"
local owasp_waf = lua_resty_waf:new()
owasp_waf:set_option("add_ruleset", "REQUEST-901-INITIALIZATION.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-930-APPLICATION-ATTACK-LFI.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-931-APPLICATION-ATTACK-RFI.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-932-APPLICATION-ATTACK-RCE.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-933-APPLICATION-ATTACK-PHP.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-941-APPLICATION-ATTACK-XSS.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-942-APPLICATION-ATTACK-SQLI.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-943-APPLICATION-ATTACK-SESSION-FIXATION.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-944-APPLICATION-ATTACK-JAVA.conf")
owasp_waf:set_option("add_ruleset", "REQUEST-949-BLOCKING-EVALUATION.conf")

from lua-resty-waf.

Hey @GumpSun Thanks for your reply
can you let me know if this is correct

in my http block I should add below lines

init_by_lua_file /home/waf/openresty/lua-resty-waf/init.lua;
init_worker_by_lua_file /home/waf/openresty/lua-resty-waf/iworker.lua;
access_by_lua_file /home/waf/openresty/lua-resty-waf/runtime.lua;

and create init.lua and runtime.lua in /usr/local/openresty/lua-resty-waf/ directory

init.lua
require "resty.core"
local lua_resty_waf = require "resty.waf"
        lua_resty_waf.load_secrules("/usr/local/openresty/lua-resty-waf/rules/26_Apps_WordPress.conf") 
        lua_resty_waf.load_secrulres("/usr/local/openresty/lua-resty-waf/rules/00_Init_Initialization.conf")
        lua_resty_waf.load_secrules("/usr/local/openresty/lua-resty-waf/rules/18_Outgoing_FilterOther.conf")
        lua_resty_waf.load_secrules("/usr/local/openresty/lua-resty-waf/rules/19_Outgoing_FilterInFrame.conf)




runtime.lua

local lua_resty_waf = require "resty.waf"
local owasp_waf = lua_resty_waf:new()
       owasp_waf:set_option("add_ruleset", "26_Apps_WordPress.conf")
       owasp_waf:set_option("add_ruleset", "00_Init_Initialization.conf")
       owasp_waf:set_option("add_ruleset", "18_Outgoing_FilterOther.conf")
       owasp_waf:set_option("add_ruleset", "19_Outgoing_FilterInFrame.conf")

from lua-resty-waf.

@rahulbhatu It work now？These rule file 26_Apps_WordPress.conf 00_Init_Initialization.conf 18_Outgoing_FilterOther.conf 19_Outgoing_FilterInFrame.conf，You have to make sure that it's available. And I think lua-resty-waf need to be improved, eg: owasp rules file REQUEST-903.9002-WORDPRESS-EXCLUSION-RULES.conf,lua-resty-waf load that but can't use it.

from lua-resty-waf.

@GumpSun
I think the init value is nil and rule is not available when some rules are not compatible or some issue in conversion, I manually checked some rules when it gets converted and are compatible they are loaded now when I only add those rules which are available it works and init value nil error is gone.
One concern is that when I add the compatible rules i get some warning failed to convert rules and Cannot perform transform utf8toUnicode type of warning so need to check if this is problem

added this in my http block

    lua_package_path '/usr/local/openresty/site/lualib/rules/?.lua;;';
    lua_package_cpath '/usr/local/openresty/site/lualib/rules/?.lua;;';


    init_by_lua_block  {
        local lua_resty_waf = require "resty.waf" 
      
        local ok, errs = pcall(function()
        lua_resty_waf.load_secrules("/usr/local/openresty/site/lualib/rules/18_Outgoing_FilterOther.conf")
        lua_resty_waf.load_secrules("/usr/local/openresty/site/lualib/rules/19_Outgoing_FilterInFrame.conf")     
        lua_resty_waf.load_secrules("/usr/local/openresty/site/lualib/rules/26_Apps_WordPress.conf")
       end)

       if errs then
            for i = 1, #errs do
                ngx.log(ngx.ERR, errs[i])
            end
        end
    }

and below config in my server block i am ignoring default given rules.

WAF settings
    
    access_by_lua_block {
                local lua_resty_waf = require "resty.waf"
                local waf = lua_resty_waf:new()
        
                waf:set_option("event_log_request_body", true)
                waf:set_option("event_log_request_headers", true)                   
                waf:set_option("debug", true)
                waf:set_option("mode", "ACTIVE")         
                waf:set_option("debug_log_level", ngx.DEBUG)
        	waf:set_option("add_ruleset", "18_Outgoing_FilterOther.conf")
		 waf:set_option("add_ruleset", "19_Outgoing_FilterInFrame.conf")
                waf:set_option("add_ruleset", "27_Apps_WPPlugin.conf")
         
                     

                waf:set_option("ignore_ruleset", "11000_whitelist")
                waf:set_option("ignore_ruleset", "20000_http_violation")
                waf:set_option("ignore_ruleset", "21000_http_anomaly")
                waf:set_option("ignore_ruleset", "35000_user_agent")
                waf:set_option("ignore_ruleset", "40000_generic_attack")
                waf:set_option("ignore_ruleset", "41000_sqli")
                waf:set_option("ignore_ruleset", "42000_xss")
                waf:set_option("ignore_ruleset", "90000_custom")
                waf:set_option("ignore_ruleset", "99000_scoring")
                waf:set_option("info", "true")
                waf:set_option("event_log_periodic_flush", 30)
                waf:set_option("event_log_buffer_size", 128)

                waf:set_option("event_log_ngx_vars", "request_id")
                waf:set_option("event_log_ngx_vars", "server_port")
                waf:set_option("event_log_request_arguments", true)
         
                waf:set_option("allow_unknown_content_types", true)
                waf:set_option("event_log_target", "file")
                waf:set_option("event_log_target_path", "/var/log/waf/eve.log")
                waf:set_option("process_multipart_body", true)
                waf:set_option("res_body_max_size", 1024 * 1024 * 2)
                waf:set_option("req_tid_header", false)
                waf:set_option("res_tid_header", false)
                waf:set_option("res_body_mime_types", { "text/plain", "text/html", "text/json", "application/json", "text/php", "text/plain", "text/x-php", "application/php", "application/x-php", "application/x-httpd-php", "application/x-httpd-php-source" })
               
          
                waf:exec()
        }
       
       ###############################################
       # lua resty waf usage
       ###############################################

            header_filter_by_lua_block {
                local lua_resty_waf = require "resty.waf"
                local waf = lua_resty_waf:new()
                waf:exec()
            }

            body_filter_by_lua_block {
                local lua_resty_waf = require "resty.waf"
                local waf = lua_resty_waf:new()
                waf:exec()
            }

            log_by_lua_block {
                local lua_resty_waf = require "resty.waf"
                local waf = lua_resty_waf:new()
                waf:exec()
                waf:write_log_events()
            }

And Thanks Once Again @GumpSun for taking time to look into this, I am actually new to lua and openresty concepts but really liking this.

from lua-resty-waf.

@rahulbhatu It doesn't support unicode and base64,There are still some problems of it, You can debug by adding log.

from lua-resty-waf.

@GumpSun
There are bunch of such errors
load_secrules(): /usr/local/openresty/site/lualib/resty/waf/translate.lua:738: Cannot translate operator validateByteRange
Cannot translate action multiMatch
usr/local/openresty/site/lualib/resty/waf/translate.lua:738: Cannot translate operator endswith
/usr/local/openresty/site/lualib/resty/waf/translate.lua:738: Cannot translate operator endswith

so the rules will work or it is just a warning for translation error all these errors have prefix nginx: [warn] [lua]
or will have to debug all the rules to make it work however I can see in the event log the rules are getting detected

{"timestamp":1571229582,"request_headers":{"host":"demo.host","content-type":"application\/x-www-form-urlencoded; charset=UTF-8","connection":"keep-alive","cookie":"wordpress_sec_7bace0758ea862e8d397bb7e154be972=rahulbhatu%7C1571400827%7CEdnApysdkny5T8YAK9UYvTWAwSiGdOFXzlhXkZclQCD%7C1ab3cf5649825b04bc3c11ac4ba8a143580af007d4d2b1cd5165fd466a7d8e7d; wordpress_test_cookie=WP+Cookie+check; wordpress_logged_in_7bace0758ea862e8d397bb7e154be972=rahulbhatu%7C1571400827%7CEdnApysdkny5T8YAK9UYvTWAwSiGdOFXzlhXkZclQCD%7Ca8f9453c419dc9ea61d99b2dcfcd9c2680f86d4c0b181fc0e114822f3a230975; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1571228027","sec-fetch-site":"same-origin","accept-language":"en-GB,en-US;q=0.9,en;q=0.8","referer":"https:\/\/demo.host\/wp-admin\/index.php","content-length":"111","accept-encoding":"gzip, deflate, br","x-requested-with":"XMLHttpRequest","sec-fetch-mode":"cors","user-agent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/77.0.3865.120 Safari\/537.36","origin":"https:\/\/demo.host","accept":"application\/json, text\/javascript, *\/*; q=0.01"},"id":"87afad2ecbc606ddb9d7","method":"POST","request_body":{"_nonce":"42d6b26b88","interval":"60","action":"heartbeat","screen_id":"dashboard","data[wp-auth-check]":"true","has_focus":"false"},"uri":"\/wp-admin\/admin-ajax.php","client":"103.225.132.225","alerts":[{"**msg":"COMODO WAF: Track unauthenticated request in WordPress||nil|nil|2","id":"225250","match":0}],**"uri_args":{},"ngx":{"request_id":"5311c37d5d6aea0430a0918b41b5ffa8","server_port":"443"}}

from lua-resty-waf.

@rahulbhatu It's better debug all the rules to make it work. It doesn't support multiMatch.You can check this https://github.com/p0pr0ck5/lua-resty-waf/wiki/Known-ModSecurity-Translation-Limitations.

from lua-resty-waf.

@GumpSun This will be really difficult for number of rules, I will add rules one by one to check for errors.

from lua-resty-waf.