Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 55: Use a explicit Query for fetching the Submission Type list. #71

Merged

Conversation

kaladay
Copy link

@kaladay kaladay commented Apr 29, 2024

Relates #55

The test data consisted of randomly generated data with:

  • 44075 submissions.
  • 1458591 field values.

The submission list page took ~3.5 minutes to load the Submission Type list data.
Switching to an explicit query, the submission list page took ~320 milliseconds to load the Submission Type list data.

This is an overwhelming clear improvement that shows that the use of Spring+JPA internals are highly inefficient for the Field Values and related data.
For the filters, the actual model does not need to be loaded and only the value.

The use of a JDBC query is chosen after doing tests against Postgresql and H2.
The use of a native query via @Query worked for Postgresql but H2 used clob2 and failed to convert the results into strings.
Using a literal query is chosen over a JPA query because the JPA wants to understand the model structure.
The Field Predicate is not loaded immediately available for JPA for the FieldValue JPA Repository.

Only very basic SQL is used in the query and so there should be very little SQL compatibility risks.
The biggest risk is that of the table naming structure.
This structure appears to be sufficiently consistent between Postgresql and H2 in this regard.

Spring+JPA Performance:
test_data-44075_subm-1458591_fv-jpa-Screenshot_2024-04-29_09-21-09

JDBC Performance:
test_data-44075_subm-1458591_fv-jdbc-Screenshot_2024-04-29_09-21-09

The test data consisted of randomly generated data with:
- 44075 submissions.
- 1458591 field values.

The submission list page took ~3.5 minutes to load the Submission Type list data.
Switching to an explicit query, the submission list page took ~320 milliseconds to load the Submission Type list data.

This is an overwhelming clear improvement that shows that the use of Spring+JPA internals are highly inefficient for the Field Values and related data.
For the filters, the actual model does not need to be loaded and only the value.

The use of a JDBC query is chosen after doing tests against Postgresql and H2.
The use of a native query via `@Query` worked for Postgresql but H2 used clob2 and failed to convert the results into strings.
Using a literal query is chosen over a JPA query because the JPA wants to understand the model structure.
The Field Predicate is not loaded immediately available for JPA for the FieldValue JPA Repository.

Only very basic SQL is used in the query and so there should be very little SQL compatibility risks.
The biggest risk is that of the table naming structure.
This structure appears to be sufficiently consistent between Postgresql and H2 in this regard.
@kaladay kaladay requested a review from jsavell April 29, 2024 16:05
@kaladay kaladay linked an issue Apr 29, 2024 that may be closed by this pull request
kaladay added 2 commits April 29, 2024 11:18
… changes.

I accidentally removed too much code when I cleaned up the previous commit (abb885a) to separate unrelated changes.
This adds back the changes that I forgot to commit.

This updates the CLI to use the `getAllValuesByFieldPredicateValue()` call.
The query is not a static final string and this line is no longer needed or used.
@kaladay
Copy link
Author

kaladay commented Apr 30, 2024

I have been able to confirm that this works in MySQL/MariaDB.

To get MySQL/MariaDB working, I had to alter src/main/java/org/tdl/vireo/model/Organization.java, changing:

    @ElementCollection(fetch = EAGER)
     private List<String> emails;

Into

    @LazyCollection(LazyCollectionOption.FALSE)
    @ElementCollection
     private List<String> emails;

@kaladay kaladay merged commit a5e4b93 into sprint1-staging Apr 30, 2024
0 of 2 checks passed
@jsavell jsavell mentioned this pull request Apr 30, 2024
@kaladay kaladay deleted the 55-submission_type_filter-performance_follow_up branch May 31, 2024 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add custom filter for submission type by list
2 participants