molecule testing, multiple fixes, fingerprinting docs

Fingerprinting.md

@@ -0,0 +1,24 @@
+# HAProxy Fingerprinting
+
+We are only dealing with server-side fingerprinting.
+
+This kind of fingerprinting only has a very limited set of information to work with - including:
+
+* TCP data
+* SSL data
+* HTTP data
+
+If you want to have a fingerprint that is unique for each client that connects, you might want to look into implementing [Client-side fingerprinting](https://wiki.superstes.eu/en/latest/1/infra/waf.html#client-side-fingerprint) in your web application frontend.
+
+Check out my [WAF Docs](https://wiki.superstes.eu/en/latest/1/infra/waf.html) for more details.
+
+## SSL Fingerprint (JA3)
+
+This fingerprint will be the same for every HTTP client. Per example: Chrome 118.1.1 will have the same one - no matter were it comes from. This can be pretty useful to track/recognize a distributed attack.
+
+You may not want to use this kind of fingerprint for blocking clients. But it can be combined with other data to limit the block-scope.
+
+If you enable `security.fingerprint_ssl` you can reference it using the variables:
+
+* `var(txn.fingerprint_ssl)` => MD5 hash of JA3 fingerprint
+* `var(txn.fingerprint_ssl_raw)` => raw JA3 fingerprint

README.md

@@ -58,6 +58,11 @@ ansible-galaxy install -r requirements.yml
 * **Package installation**
   * Repository dependencies (_minimal_)
   * HAProxy
+  * GeoIP
+    * [Lookup Service Binary](https://github.com/superstes/geoip-lookup-service)
+  * ACME
+    * [Dependencies](https://github.com/dehydrated-io/dehydrated/blob/v0.7.1/dehydrated#L261)
+    * Nginx light for challenge-response handling
 
 
 * **Configuration**
@@ -72,9 +77,6 @@ ansible-galaxy install -r requirements.yml
         * Redirect non SSL traffic to SSL
         * Logging User-Agent
         * Setting basic security-headers
-
-    * Backend
-      * HTTP mode
         * Blocking TRACE & CONNECT methods
 
 
@@ -84,11 +86,11 @@ ansible-galaxy install -r requirements.yml
       * [ACME/LetsEncrypt](https://github.com/dehydrated-io/dehydrated)
       * [GeoIP Lookups](https://github.com/superstes/haproxy-geoip)
       * Blocking of well-known Script-Bots
-      * Blocking TRACE & CONNECT methods
       * SSL Fingerprinting ([JA3](https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967/?ref=waf.ninja))
 
     * Backend
       * Sticky sessions (*use same backend )
+      * Blocking TRACE & CONNECT methods
 
 ----
 
@@ -110,19 +112,18 @@ ansible-galaxy install -r requirements.yml
     `filter_ip`, `filter_not_ip`, `filter_country`, `filter_not_country`, `filter_asn`, `filter_not_asn`
 
 
-* **Info:** A very basic user-agent based Script- & Bad-Crawler-Bot blocking can be activated for frontends and backends. Check out the [defaults](https://github.com/ansibleguy/infra_haproxy/blob/latest/defaults/main/0_hardcoded.yml) for the list of bots that are blocked.
+* **Info:** A very basic user-agent based Script- & Bad-Crawler-Bot blocking can be activated for frontends and backends. Check out the [defaults](https://github.com/ansibleguy/infra_haproxy/blob/latest/defaults/main/2_waf.yml) for the list of bots that are blocked.
 
 
 * **Info:** You can easily restrict the HTTP methods allowed on a specific frontend or backend by setting `security.restrict_methods` to true and specifying `security.allow_only_methods`
 
 
-* **Info:** If you are using [Graylog Server](https://graylog.org/products/source-available/) to gather and analyze your logs - make sure to split your HAProxy logs into fields using pipeline rules. Example: [HAProxy Community - Graylog Pipeline Rule](https://gist.github.com/superstes/a2f6c5d855857e1f10dcb51255fe08c6#haproxy-split)
+* **Info:** Check out the [Fingerprinting Docs](https://github.com/ansibleguy/infra_haproxy/blob/latest/Fingerprinting.md) for detailed information on how you might want to track clients.
 
 
-* **Info:** If you enable `fingerprint_ssl` you can reference it using the variables:
+* **Info:** If you are using [Graylog Server](https://graylog.org/products/source-available/) to gather and analyze your logs - make sure to split your HAProxy logs into fields using pipeline rules. Example: [HAProxy Community - Graylog Pipeline Rule](https://gist.github.com/superstes/a2f6c5d855857e1f10dcb51255fe08c6#haproxy-split)
+
 
-    * `var(txn.fingerprint_ssl)` => MD5 hash of JA3 fingerprint
-    * `var(txn.fingerprint_ssl_raw)` => raw JA3 fingerprint
 
 ### GeoIP
 
@@ -289,9 +290,9 @@ haproxy:
         default: 'http-request redirect code 301 location https://github.com/ansibleguy'
 
   # GENERAL
-  stats:  # enable stats http page
-    http: true
-    bind: '127.0.0.1:8404'
+  stats:
+    enable: true  # enable stats http listener
+    bind: '127.0.0.1:8404'  # default
 
   geoip:
     enable: true

defaults/main/0_hardcoded.yml

@@ -39,7 +39,6 @@ HAPROXY_HC:
     acme_script: "https://github.com/dehydrated-io/dehydrated/releases/download/v{{ version_dehydrated }}/dehydrated-{{ version_dehydrated }}.tar.gz"
 
   valid_geoip_providers: ['ipinfo', 'maxmind']
-  geoip_lookup_port: 10069
   user_geoip: 'haproxy-geoip'
   service_geoip_lookup: 'haproxy-geoip-lookup'
   service_geoip_update: 'haproxy-geoip-update'
@@ -60,56 +59,3 @@ HAPROXY_HC:
   acme_reload_timer: '*-*-* 03:00:00'
   service_acme: 'haproxy-acme'
   service_acme_reload: 'haproxy-acme-reload'
-
-  # see also: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-user-agents.list
-  user_agents:
-    # Well-known user-agents used by scripting languages for very basic security-filtering
-    # matching is done case-insensitive
-    script:
-      # NOTE: empty user-agent is also matched
-      full: []
-
-      # NOTE: these are sub-strings inside the user-agent header
-      sub:
-        # cli tools
-        - 'curl'
-        - 'wget'
-        - 'Apache-HttpClient'
-        - 'nmap'
-        - 'Metasploit'
-        # automation tools
-        - 'headless'
-        - 'cypress'
-        # golang
-        - 'go-http-client'
-        - 'zgrab'
-        # python
-        - 'python'
-        - 'httpx'
-        - 'httpcore'
-        - 'aiohttp'
-        - 'httputil'
-        - 'urllib'
-        # php
-        - 'GuzzleHttp'
-        - 'phpcrawl'
-        - 'Zend_Http_Client'
-        - 'Wordpress'
-        - 'Symfony-HttpClient'
-        # others
-        - 'cpp-httplib'  # c++
-        - 'java'
-        - 'perl'
-        - 'axios'  # JS
-        - 'ruby'
-
-    bad_crawlers:
-      full: []
-      sub:
-        - 'spider'
-        - 'test-bot'
-        - 'tiny-bot'
-        - 'fidget-spinner-bot'
-        - 'download'
-        # python
-        - 'scrapy'

defaults/main/1_main.yml

@@ -9,10 +9,11 @@ defaults_haproxy:
   version: '2.8'
   frontends: {}
   backends: {}
+  waf: {}
 
   # see: https://www.haproxy.com/blog/exploring-the-haproxy-stats-page
   stats:
-    http: false
+    enable: false  # enable http listener for haproxy stats
     bind: '127.0.0.1:8404'
     refresh: '10s'
     admin: false
@@ -21,18 +22,20 @@ defaults_haproxy:
     pwd: 'monitor'
     realm: 'Authorized Personal Only'
 
-  acme:
+  acme:  # see also: https://github.com/dehydrated-io/dehydrated/blob/master/docs/examples/config
     enable: false
     email:
     ocsp_must_staple: false
     ocsp_fetch: false
     manage_challenge: true  # set-up nginx-light to handle acme challenge-responses
     challenge_port: 8405  # port the webserver for challenge-responses listens on
+    ca: 'letsencrypt'
     domains: []
 
   geoip:
     enable: false
     manage_db: true
+    lookup_port: 8406
     provider: 'ipinfo'
     token:
 
@@ -72,6 +75,7 @@ defaults_haproxy:
 
 defaults_frontend:
   mode: 'http'
+  http2: true  # appends 'alpn h2,http/1.1' to bind
   bind: ['[::]:80 v4v6']
   # ipv4 only: ':80'
   # ssl with acme enabled: ':443 ssl'
@@ -83,25 +87,25 @@ defaults_frontend:
   ssl_redirect: true
   security:
     headers: true
-    fingerprint_ssl: true  # create and log the JA3 fingerprint of clients
+    fingerprint_ssl: false  # create and log the JA3 fingerprint of clients
 
     restrict_methods: false
     allow_only_methods: ['HEAD', 'GET', 'POST']
     # if 'restrict_methods' is disabled - this will still deny 'TRACE' & 'CONNECT' as they might open your server/services up to attacks
-    deny_dangerous_methods: false
+    deny_dangerous_methods: true
 
     block_script_bots: false
     block_bad_crawler_bots: false
-    block_status_code: 425
 
   log:
     user_agent: true
 
   geoip:
     enable: false
+    # what attributes to look-up:
     country: true
     asn: true
-    asn_name: false
+    as_name: false
 
   lines: {}  # raw config lines to add - section to lines mapping to make resulting config human-readable
 
@@ -141,11 +145,10 @@ defaults_backend:
     restrict_methods: false
     allow_only_methods: ['HEAD', 'GET', 'POST']
     # if 'restrict_methods' is disabled - this will still deny 'TRACE' & 'CONNECT' as they might open your server/services up to attacks
-    deny_dangerous_methods: true
+    deny_dangerous_methods: false
 
     block_script_bots: false
     block_bad_crawler_bots: false
-    block_status_code: 425
 
   # for health-checks see: https://www.haproxy.com/blog/how-to-enable-health-checks-in-haproxy
   #   more complex ones should be implemented by supplying the raw config-lines

defaults/main/2_waf.yml

@@ -0,0 +1,59 @@
+---
+
+defaults_waf:
+  block_code: 425
+
+  # todo: search parameters and body of http requests for SQLi/Path-Traversal/XSS/...
+
+  # see also: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/_generator_lists/bad-user-agents.list
+  user_agents:
+    # Well-known user-agents used by scripting languages for very basic security-filtering
+    # matching is done case-insensitive
+    script:
+      # NOTE: empty user-agent is also matched
+      full: [ ]
+
+      # NOTE: these are sub-strings inside the user-agent header
+      sub:
+        # cli tools
+        - 'curl'
+        - 'wget'
+        - 'Apache-HttpClient'
+        - 'nmap'
+        - 'Metasploit'
+        # automation tools
+        - 'headless'
+        - 'cypress'
+        # golang
+        - 'go-http-client'
+        - 'zgrab'
+        # python
+        - 'python'
+        - 'httpx'
+        - 'httpcore'
+        - 'aiohttp'
+        - 'httputil'
+        - 'urllib'
+        # php
+        - 'GuzzleHttp'
+        - 'phpcrawl'
+        - 'Zend_Http_Client'
+        - 'Wordpress'
+        - 'Symfony-HttpClient'
+        # others
+        - 'cpp-httplib'  # c++
+        - 'java'
+        - 'perl'
+        - 'axios'  # JS
+        - 'ruby'
+
+    bad_crawlers:
+      full: [ ]
+      sub:
+        - 'spider'
+        - 'test-bot'
+        - 'tiny-bot'
+        - 'fidget-spinner-bot'
+        - 'download'
+        # python
+        - 'scrapy'

defaults/main/9_merge.yml

@@ -1,3 +1,4 @@
 ---
 
 HAPROXY_CONFIG: "{{ defaults_haproxy | combine(haproxy, recursive=true, list_merge='prepend') }}"
+HAPROXY_WAF: "{{ defaults_waf | combine(HAPROXY_CONFIG.waf, recursive=true) }}"

handlers/main.yml

@@ -9,3 +9,8 @@
   ansible.builtin.systemd:
     name: 'haproxy.service'
     state: reloaded
+
+- name: HAProxy-geoip-restart
+  ansible.builtin.systemd:
+    name: "{{ HAPROXY_HC.service_geoip_lookup }}"
+    state: restarted

molecule/default/Usage.md

@@ -0,0 +1,10 @@
+# Usage
+
+Check out the [Molecule Tutorial](https://github.com/ansibleguy/ansible_tutorial/blob/main/99/Molecule.md) on how to get started!
+
+# Running
+
+```bash
+cd roles/ansibleguy.infra_haproxy
+molecule test
+```