Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: ctlstore oom #126

Closed
wants to merge 5 commits into from
Closed

test: ctlstore oom #126

wants to merge 5 commits into from

Conversation

preetyp
Copy link

@preetyp preetyp commented Oct 10, 2023

I have opened this PR for experimental purposes to resolve the OOM issue in warehouses. Once I get approval for my approach, I will add other things such as test cases, and this PR is currently in draft mode.
Jira id : https://segment.atlassian.net/browse/CONSENT-139

Purpose :
in tsub library, currently we are using

func (reader *LDBReader) GetRowsByKeyPrefix(ctx context.Context, familyName string, tableName string, key ...interface{}) (*Rows, error) {
function to load all rules based on key array(which is array of primary keys, 1st entry represents scope, 2nd entry represents target_type and 3rd represents target_id), (we are using 1st entry for scope as a key)
and query is
SELECT * FROM tsub_store___rules_materialized_2 WHERE scope = ?
but this is loading ~83k data in cache map for scope=destinations. and in every ttl expiry, we are again loading all scope=destinations based rules.
which is causing OOM in warehouses.
To avoid this issue we want to load only warehouseId specific rules only in cache map for warehouses. which will require targetId like %destinationId% query.
Because this method only provides exact match using = operator. So, introduced a new method for like query.
so, now new function will generate & execute below query
SELECT * FROM tsub_store___rules_materialized_2 WHERE target_id LIKE %destinationId%

Summary
mainly memory consumption is in storing and deserializing process of cacheRules(..) function in tsub's ctlstore.go

we are currently loading ~83K rows from db
storing them and deserializing each rows in every ttl expiry

this is the main cause of OOM in warehouses, other storage destinations are fine with it

if i reduce this number from 83K to ~2 digits number by not loading all rules based on scope but by loading all rules based on warehouseId..
then no OOM in warehouses as well

@preetyp preetyp requested a review from a team as a code owner October 10, 2023 10:06
@preetyp preetyp added the Don't Merge This is only for performance testing and to resolve OOM issue for consent feature in warehouses label Oct 10, 2023
@preetyp preetyp marked this pull request as draft October 10, 2023 20:01
@erikdw
Copy link
Collaborator

erikdw commented Jun 17, 2024

Gonna close this since we aren't doing this for now

@erikdw erikdw closed this Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Don't Merge This is only for performance testing and to resolve OOM issue for consent feature in warehouses
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants