fix!: reject duplicate submissions #5047
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rajpatel24
force-pushed
the
862-reject_duplicate_submissions
branch
from
28eefe5 to
039ed95
Compare
rajpatel24
changed the title
rajpatel24
force-pushed
the
862-reject_duplicate_submissions
branch
from
039ed95 to
6a177da
Compare
noliveleger
changed the base branch from
beta
to
kobocat-django-app-part-2-refactor-mock-deployment-backend
# Conflicts: # kobo/apps/openrosa/apps/logger/exceptions.py # kobo/apps/openrosa/apps/logger/models/instance.py # kobo/apps/openrosa/apps/logger/models/xform.py # kobo/apps/openrosa/apps/logger/xform_instance_parser.py # kobo/apps/openrosa/libs/utils/logger_tools.py
noliveleger
changed the base branch from
kobocat-django-app-part-2-refactor-mock-deployment-backend
to
beta-refactored
This was referenced Aug 27, 2024
noliveleger
requested changes
Aug 27, 2024
kobo/apps/openrosa/apps/api/tests/viewsets/test_xform_submission_api.py
Outdated
Show resolved
Hide resolved
noliveleger
reviewed
Aug 27, 2024
…-reject_duplicate_submissions
rajpatel24
force-pushed
the
862-reject_duplicate_submissions
branch
2 times, most recently
from
bf597d5 to
2958052
Compare
- Add test case for duplicate submission with an attachment - Improve logic to extract UUID from xml - Add logic to reject submission without UUID
rajpatel24
force-pushed
the
862-reject_duplicate_submissions
branch
from
2958052 to
f3c89f6
Compare
noliveleger
requested changes
Sep 3, 2024
noliveleger
requested changes
Sep 3, 2024
noliveleger
requested review from
bufke,
jnm,
magicznyleszek and
Guitlle
as code owners
rajpatel24
force-pushed
the
862-reject_duplicate_submissions
branch
from
a1284fd to
578b2be
Compare
noliveleger
reviewed
Jan 8, 2025
Comment on lines
+222
to
+237
| int_lock = int.from_bytes( | ||
| hashlib.shake_128( | ||
| f'{xform_id}!!{submission_uuid}!!{xml_hash}'.encode() | ||
| ).digest(7), 'little' | ||
| ) | ||
| acquired = False |
There was a problem hiding this comment.
@rajpatel24 please write a comment to explain why we use this thing.
Guitlle
approved these changes
Jan 8, 2025
rajpatel24
force-pushed
the
862-reject_duplicate_submissions
branch
from
578b2be to
b0a4208
Compare
…lbox/kpi into 862-reject_duplicate_submissions
Indeed, but this PR predates the new PR workflow. |
…lbox/kpi into 862-reject_duplicate_submissions
noliveleger
approved these changes
Jan 13, 2025
There was a problem hiding this comment.
@rajpatel24 I'll let you have the honors of merging this PR.
FYI: I pushed one tiny commit for the management command to fix it to avoid another back and forth.
noliveleger
changed the title
noliveleger
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implemented logic to detect and reject duplicate submissions.
Description
We have identified a race condition in the submission processing that causes duplicate submissions with identical UUIDs and XML hashes. This issue is particularly problematic under conditions with multiple remote devices submitting forms simultaneously over unreliable networks.
To address this issue, a PR has been raised with the following proposed changes:
Race Condition Resolution: A locking mechanism has been added to prevent the race condition when checking for existing instances and creating new ones. This aims to eliminate duplicate submissions.
UUID Enforcement: Submissions without a UUID are now explicitly disallowed. This ensures that every submission is uniquely identifiable and further mitigates the risk of duplicate entries.
Introduction of
root_uuid:To ensure a consistent submission UUID throughout its lifecycle and prevent duplicate submissions with the same UUID, a new
root_uuidcolumn has been added to theInstancemodel with a unique constraint (root_uuidperxform).If the
<meta><rootUuid>is present in the submission XML, it is stored in theroot_uuidcolumn.If
<meta><rootUuid>is not present, the value from<meta><instanceID>is used instead.This approach guarantees that the
root_uuidremains constant across the lifecycle of a submission, providing a reliable identifier for all instances.UUID Handling Improvement: Updated the logic to strip only the
uuid:prefix while preserving custom, non-UUID ID schemes (e.g., domain.com:1234). This ensures compliance with the OpenRosa spec and prevents potential ID collisions with custom prefixes.Error Handling:
These changes should improve the robustness of the submission process and prevent both race conditions and invalid submissions.
Notes
Related issues
Supersedes kobotoolbox/kobocat#876
and kobotoolbox/kobocat#859