ISSUE-2269: tests for access log transmission to Clickhouse server #749
Conversation
t_clickhouse/test_clickhouse_logs.py
Outdated
| self.assertEqual(wrk_total_requests, dmesg_logs_count) | ||
|
|
||
| def test_all_logs_with_reload(self): | ||
| client = self.prepare_client() |
There was a problem hiding this comment.
Maybe you should move these stress tests to t_stress directory? Or you can just add access_log to exists tests.
There was a problem hiding this comment.
What do you mean under " add access_log to exists tests" ?
There was a problem hiding this comment.
We have some stress tests in t_stress directory. You can add access_log directive to these tests. I don't think that we need a new stress tests.
# Conflicts: # tests_disabled.json
| ) | ||
|
|
||
| record = self.loggers.clickhouse.access_log_last_message() | ||
| self.assertEqual(record.status, 500) |
There was a problem hiding this comment.
Sometimes I receive error here:
======================================================================
ERROR: test_clickhouse_record_data_with_post (t_clickhouse.test_clickhouse_logs.TestClickHouseLogsCorrectnessDataPostRequest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/tempesta-test/t_clickhouse/test_clickhouse_logs.py", line 283, in test_clickhouse_record_data_with_post
self.assertEqual(record.status, 500)
AttributeError: 'NoneType' object has no attribute 'status'
----------------------------------------------------------------------
It is unstable test, we should fix this now. Probably should you use find method here?
| @@ -44,7 +44,7 @@ as root: | |||
|
|
|||
| - Python version till 3.11 is supported, version **3.12 is NOT supported** | |||
| (we use [asyncore](https://docs.python.org/3.11/library/asyncore.html) that was removed in 3.12) | |||
|
|
|||
| - ClickHouse Database 25 (Optional, used for storing logs in the database). See installation [here](https://clickhouse.com/docs/en/install#quick-install) | |||
There was a problem hiding this comment.
Please add info about creating the table into clickhouse here or TempestaFW wiki.
There was a problem hiding this comment.
https://github.com/tempesta-tech/tempesta/wiki/Handling-clients#clickhouse
It's already exists in the wiki
| if not messages: | ||
| return None | ||
|
|
||
| return messages[-1] |
There was a problem hiding this comment.
Optional:
| if not messages: | |
| return None | |
| return messages[-1] | |
| return messages[-1] or None |
There was a problem hiding this comment.
>>> ll = []
>>> ll[-1] or None
Traceback (most recent call last):
File "<python-input-1>", line 1, in <module>
ll[-1] or None
~~^^^^
IndexError: list index out of range| if isinstance(self.log, bytes): | ||
| return AccessLogLine.parse_all(self.log.decode()) | ||
|
|
||
| return AccessLogLine.parse_all(self.log) |
There was a problem hiding this comment.
optional:
| if isinstance(self.log, bytes): | |
| return AccessLogLine.parse_all(self.log.decode()) | |
| return AccessLogLine.parse_all(self.log) | |
| return AccessLogLine.parse_all(self.log.decode() if isinstance(self.log, bytes) else self.log) |
There was a problem hiding this comment.
I prefer to leave the code as it is because, to me, it looks more readable. However, I believe there’s a valid reason to create an issue regarding the invalid type of self.log, as it sometimes becomes a string or bytes. I wanted to address this immediately, but it would require verifying a lot of code, which makes it risky to do within this PR.
| self.loggers.dmesg.update() | ||
| self.assertEqual(self.loggers.dmesg.access_log_records_count(), 1) |
There was a problem hiding this comment.
You should find method here. It is unstable test.
| self.send_simple_request(client) | ||
| self.assertEqual(self.loggers.dmesg.access_log_records_count(), 0) | ||
|
|
||
| tempesta = self.get_tempesta() | ||
| tempesta.stop() | ||
|
|
||
| pattern = r".*Starting daemon.*Daemon started.*Stopping daemon.*Daemon stopped.*" | ||
| self.assertTrue(self.loggers.clickhouse.find(pattern)) |
There was a problem hiding this comment.
| self.send_simple_request(client) | |
| self.assertEqual(self.loggers.dmesg.access_log_records_count(), 0) | |
| tempesta = self.get_tempesta() | |
| tempesta.stop() | |
| pattern = r".*Starting daemon.*Daemon started.*Stopping daemon.*Daemon stopped.*" | |
| self.assertTrue(self.loggers.clickhouse.find(pattern)) | |
| self.send_simple_request(client) | |
| self.assertTrue(self.loggers.clickhouse.find("200")) # or other pattern for the requst | |
| self.loggers.dmesg.update() | |
| self.assertEqual(self.loggers.dmesg.access_log_records_count(), 0) | |
You must call update method for dmesg before access_log_* method or the log will be empty.
No description provided.