lua-only option, removed shell-backend

README.md

@@ -6,19 +6,36 @@ Data linking requests to its origin country and ASN/ISP can be very useful when
 
 This allows you also to handle requests from specific countries and ASNs (p.e. datacenters/hosting providers) differently than others.
 
+NOTE: This functionality is covered by the [HAProxy Enterprise Maxmind-Module](https://www.haproxy.com/documentation/hapee/latest/load-balancing/geolocation/maxmind/)! Only use this implementation if you are limited to the community edition.
+
 ## Topology
 
+You can implement this in two ways:
+
+* Use a custom backend to do this lookups
+* UNTESTED: Use the [resty-maxminddb LUA library](https://raw.githubusercontent.com/anjia0532/lua-resty-maxminddb/master/lib/resty/maxminddb.lua) to query the MMDB databases directly from LUA
+
+### Lookup via Backend
+
 1. Request hits HAProxy
 
 2. HAProxy calls LUA script to delegate GeoIP-database lookup
 
-   I would prefer to use [io.popen](https://www.lua.org/manual/5.1/manual.html#pdf-io.popen) so LUA directly executes the query, but this [is blocked by HAProxy](https://discourse.haproxy.org/t/haproxy-1-8-update-to-2-7-3-lua-issues/8454)
-
-3. LUA calls a minimal web-service on localhost that queries the GeoIP-database
+3. LUA calls a minimal web-service on localhost that queries the GeoIP-database(s)
 
    In this case we use a basic Python3 HTTP-Server
 
-<img src="https://raw.githubusercontent.com/superstes/haproxy-geoip-lua/latest/topology.svg" width=300>
+<img src="https://raw.githubusercontent.com/superstes/haproxy-geoip/latest/topology.svg" width=300>
+
+
+I might add a Golang backend later on.
+
+### Lookup via Library
+
+1. Request hits HAProxy
+
+2. HAProxy calls LUA script for querying the GeoIP-database(s)
+
 
 ----
 
@@ -48,40 +65,45 @@ You will have to download some MMDB GeoIP databases.
 Per example from [ipinfo.io](https://ipinfo.io/account/data-downloads) or [maxmind](https://maxmind.com)!
 
 
-### Lookup-Backend
+### Lookup
 
-This repository shows two different backend-implementations.
+#### via Backend
 
-One calls a shell-util, the other one uses [a library](https://github.com/maxmind/MaxMind-DB-Reader-python).
-
-#### Shell-Util
-
-To query the MMDB databases, you will have to install the `mmdblookup` util:
+To query the MMDB databases, you will have to install the [maxminddb python-module](https://github.com/maxmind/MaxMind-DB-Reader-python):
 
 ```bash
-apt install mmdb-bin
+python3 -m pip install maxminddb
 ```
 
-You will have to update the paths to your database-files in the `geoip_lookup_backend_shell.py` file!
+You will have to update the paths to your database-files in the `backend/geoip_lookup_backend.py` file!
 
-#### Library
+You need to use the `lua/geoip_lookup_w_backend.lua` script.
 
-To query the MMDB databases, you will have to install the `maxminddb` python-module:
+#### via Library
 
-```bash
-python3 -m pip install maxminddb
-```
+WARNING: UNTESTED
 
-You will have to update the paths to your database-files in the `geoip_lookup_backend_lib.py` file!
+To query the MMDB databases, you will have to install the [resty-maxminddb LUA library](https://raw.githubusercontent.com/anjia0532/lua-resty-maxminddb/master/lib/resty/maxminddb.lua) and its dependencies.
 
+You need to use the `lua/geoip_lookup_w_lib.lua` script.
 
 ----
 
 ## Run
 
+### With LUA-Library
+
+```bash
+# initialize the haproxy map(s)
+touch /tmp/haproxy_geoip_country.map
+# start haproxy
+haproxy -W -f haproxy_example.cfg
+```
+
+### With Lookup-Backend
 ```bash
 # start the web-service
-python3 geoip_lookup_backend.py &
+python3 backend/geoip_lookup.py &
 # initialize the haproxy map(s)
 touch /tmp/haproxy_geoip_country.map
 # start haproxy
@@ -104,30 +126,24 @@ At least IPInfo OR MaxMind databases need to exist!
 ```bash
 cd test
 bash test.sh
-> 
+>
+> CLEANUP
+>
 > WARN: UNABLE TO TEST MaxMind databases as they are missing!
-> 
+>
 > STARTING HAPROXY
-> 
-> TESTING BACKEND with Lookup-Util
-> LINKING IPInfo databases
-> 127.0.0.1 - - [20/Nov/2023 19:10:14] "GET /?lookup=country&ip=1.1.1.1 HTTP/1.1" 200 -
-> 127.0.0.1 - - [20/Nov/2023 19:10:14] "GET /?lookup=continent&ip=1.1.1.1 HTTP/1.1" 200 -
-> 127.0.0.1 - - [20/Nov/2023 19:10:14] "GET /?lookup=asn&ip=1.1.1.1 HTTP/1.1" 200 -
-> 127.0.0.1 - - [20/Nov/2023 19:10:14] "GET /?lookup=asname&ip=1.1.1.1 HTTP/1.1" 200 -
-> REQUEST TIMES: 0.03 => 0.00 (cached)
-> 
-> TESTING BACKEND with Lookup-Lib
+>
+> TESTING with PYTHON-BACKEND
 > LINKING IPInfo databases
-> 127.0.0.1 - - [20/Nov/2023 19:10:22] "GET /?lookup=country&ip=1.1.1.1 HTTP/1.1" 200 -
-> 127.0.0.1 - - [20/Nov/2023 19:10:22] "GET /?lookup=continent&ip=1.1.1.1 HTTP/1.1" 200 -
-> 127.0.0.1 - - [20/Nov/2023 19:10:22] "GET /?lookup=asn&ip=1.1.1.1 HTTP/1.1" 200 -
-> 127.0.0.1 - - [20/Nov/2023 19:10:22] "GET /?lookup=asname&ip=1.1.1.1 HTTP/1.1" 200 -
-> REQUEST TIMES: 0.03 => 0.00 (cached)
+> 127.0.0.1 - - [22/Nov/2023 21:21:00] "GET /?lookup=country&ip=1.1.1.1 HTTP/1.1" 200 -
+> 127.0.0.1 - - [22/Nov/2023 21:21:00] "GET /?lookup=continent&ip=1.1.1.1 HTTP/1.1" 200 -
+> 127.0.0.1 - - [22/Nov/2023 21:21:00] "GET /?lookup=asn&ip=1.1.1.1 HTTP/1.1" 200 -
+> 127.0.0.1 - - [22/Nov/2023 21:21:00] "GET /?lookup=asname&ip=1.1.1.1 HTTP/1.1" 200 -
+> REQUEST TIMES: 0.01 => 0.00 (cached)
 >
 > STOPPING HAPROXY
 >
 > FINISHED - exiting
 ```
 
-Feel free to [contribute more test-cases](https://github.com/superstes/haproxy-geoip-lua/blob/latest/test/test_requests.sh)!
+Feel free to [contribute more test-cases](https://github.com/superstes/haproxy-geoip/blob/latest/test/requests.sh)!

geoip_lookup_backend_lib.py → backend/geoip_lookup.py

@@ -10,22 +10,44 @@
 
 PORT = 6970
 
-# https://ipinfo.io/account/data-downloads
+# for data schema see:
+#   ipinfo: https://github.com/ipinfo/sample-database
+#   maxmind: https://github.com/maxmind/MaxMind-DB/tree/main/source-data
+
+# ipinfo - https://ipinfo.io/account/data-downloads
 DATABASES = {
     'country': {'file': '/tmp/country.mmdb', 'attr': 'country', 'fallback': '00'},
     'continent': {'file': '/tmp/country.mmdb', 'attr': 'continent', 'fallback': '00'},
+    'city': {'file': '/tmp/city.mmdb', 'attr': 'city', 'fallback': '-'},
     'asn': {'file': '/tmp/asn.mmdb', 'attr': 'asn', 'fallback': '0'},
     'asname': {'file': '/tmp/asn.mmdb', 'attr': 'name', 'fallback': '-'},
 }
 
+# maxmind
+# DATABASES = {
+#     'country': {'file': '/tmp/country.mmdb', 'attr': 'country.iso_code', 'fallback': '00'},
+#     'continent': {'file': '/tmp/country.mmdb', 'attr': 'continent.code', 'fallback': '00'},
+#     'city': {'file': '/tmp/city.mmdb', 'attr': 'city.names.en', 'fallback': '-'},
+#     'asn': {'file': '/tmp/asn.mmdb', 'attr': 'autonomous_system_number', 'fallback': '0'},
+#     'asname': {'file': '/tmp/asn.mmdb', 'attr': 'autonomous_system_organization', 'fallback': '-'},
+# }
+
 
 def _lookup_mmdb(db: dict, ip: str) -> str:
     try:
         if not Path(db['file']).is_file():
             return db['fallback']
 
         with open_database(db['file']) as db_reader:
-            return db_reader.get(ip)[db['attr']]
+            data = db_reader.get(ip)
+            for attr in db['attr'].split('.'):
+                if attr in data:
+                    data = data[attr]
+
+                else:
+                    return db['fallback']
+
+            return data
 
     except (RuntimeError, KeyError):
         return db['fallback']

geoip_lookup.lua → lua/geoip_lookup_w_backend.lua

@@ -1,38 +1,37 @@
 
 local function http_request(lookup, src)
     local s = core.tcp()
-    s:connect("127.0.0.1:6970")
-    s:send("GET /?lookup=" .. lookup .. "&ip=" .. src .. " HTTP/1.1\r\n\r\n")
+    s:connect('127.0.0.1:6970')
+    s:send('GET /?lookup=' .. lookup .. '&ip=' .. src .. ' HTTP/1.1\r\n\r\n')
     while true do
         local line = s:receive('*l')
         if not line then break end
         if line == '' then break end
     end
     local res_body = s:receive('*a')
     if res_body == nil then
-        return "00"
+        return '00'
     end
     return res_body
 end
 
 local function lookup_geoip_country(txn)
-    country_code = http_request("country", txn.f:src())
-    txn:set_var('txn.geoip_country', country_code)
-
+    country_code = http_request('country', txn.f:src())
+    txn:set_var('txn.geoip_country', country_code)    
 end
 
 local function lookup_geoip_continent(txn)
-    continent_code = http_request("continent", txn.f:src())
+    continent_code = http_request('continent', txn.f:src())
     txn:set_var('txn.geoip_continent', continent_code)
 end
 
 local function lookup_geoip_asn(txn)
-    asn = http_request("asn", txn.f:src())
+    asn = http_request('asn', txn.f:src())
     txn:set_var('txn.geoip_asn', asn)
 end
 
 local function lookup_geoip_asname(txn)
-    asname = http_request("asname", txn.f:src())
+    asname = http_request('asname', txn.f:src())
     txn:set_var('txn.geoip_asname', asname)
 end