Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Xinyual/add tokenizer and sparse encoding #1

Open
wants to merge 69 commits into
base: addTokenizerAndSparseEncoding
Choose a base branch
from
Open

Xinyual/add tokenizer and sparse encoding #1

wants to merge 69 commits into from

Conversation

xinyual
Copy link
Owner

@xinyual xinyual commented Sep 12, 2023

Description

[Describe what this change achieves]

Issues Resolved

[List any issues this PR will resolve]

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@xinyual
add tokenizer and sparse encoding
ba553c4 
Signed-off-by: xinyual <[email protected]>
@xinyual
add tokenizer and sparse encoding
578274f 
Signed-off-by: xinyual <[email protected]>
@xinyual
add tokenizer and sparse encoding
c2ffd7a 
Signed-off-by: xinyual <[email protected]>
@xinyual
add tokenizer and sparse encoding
ed8832d 
Signed-off-by: xinyual <[email protected]>
@xinyual
add tokenizer and sparse encoding
258f2cb 
Signed-off-by: xinyual <[email protected]>
@xinyual
remove special token
e9da84e 
Signed-off-by: xinyual <[email protected]>
@xinyual
add filter
ce18800 
Signed-off-by: xinyual <[email protected]>
@xinyual
try empty model
761b2d3 
Signed-off-by: xinyual <[email protected]>
@xinyual
remove warm up
6e6cd83 
Signed-off-by: xinyual <[email protected]>
@xinyual
try empty model
6618845 
Signed-off-by: xinyual <[email protected]>
@xinyual
add block
1b919d2 
Signed-off-by: xinyual <[email protected]>
@xinyual
add log
4f70e4a 
Signed-off-by: xinyual <[email protected]>
@xinyual
add log
c3e4186 
Signed-off-by: xinyual <[email protected]>
@xinyual
add log
5ccc350 
Signed-off-by: xinyual <[email protected]>
@xinyual
remove log
73f0329 
Signed-off-by: xinyual <[email protected]>
@xinyual
remove pt file detect
3b0b7d3 
Signed-off-by: xinyual <[email protected]>
@xinyual
add log
7b98bf3 
Signed-off-by: xinyual <[email protected]>
@xinyual
add functionName pipeline
13e6d77 
Signed-off-by: xinyual <[email protected]>
@xinyual
remove verify log
9daeb96 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip special token in sparse encoding
e000f22 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip omit tokenize config
c435a89 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip omit tokenize config-change warm up logic
8a7d19e 
Signed-off-by: xinyual <[email protected]>
@xinyual
reArch
39449fc 
Signed-off-by: xinyual <[email protected]>
@xinyual
deduplicate
a115e19 
Signed-off-by: xinyual <[email protected]>
@xinyual
omit ml config in sparse encoding
846f295 
Signed-off-by: xinyual <[email protected]>
@xinyual
add null config in warm up
564f1e6 
Signed-off-by: xinyual <[email protected]>
@xinyual
fix original test
a1b9351 
Signed-off-by: xinyual <[email protected]>
@xinyual
add tokenize ut half
75de44c 
Signed-off-by: xinyual <[email protected]>
@xinyual
fix sparse encoding bug
a381d64 
Signed-off-by: xinyual <[email protected]>
@xinyual
add UT for sparse encoding and tokenize
ab745e2 
Signed-off-by: xinyual <[email protected]>
@xinyual
change tokenize way
23231bc 
Signed-off-by: xinyual <[email protected]>
@xinyual
reArch add textEmbedding model
0b1e206 
Signed-off-by: xinyual <[email protected]>
@xinyual
add tokenize logic
4ded8d6 
Signed-off-by: xinyual <[email protected]>
@xinyual
add abstract
c057977 
Signed-off-by: xinyual <[email protected]>
@xinyual
clear code
3abb2df 
Signed-off-by: xinyual <[email protected]>
@xinyual
fix it class
540df26 
Signed-off-by: xinyual <[email protected]>
@xinyual
fix it class
5dc8197 
Signed-off-by: xinyual <[email protected]>
@xinyual
add IT file
5dbc4b7 
Signed-off-by: xinyual <[email protected]>
@xinyual
reformulate
9ae26f2 
Signed-off-by: xinyual <[email protected]>
@xinyual
reformulate remote inference
fe20404 
Signed-off-by: xinyual <[email protected]>
@xinyual
reformulate remote inference
e8ee101 
Signed-off-by: xinyual <[email protected]>
@xinyual
reformulate remote inference json and array
8a67625 
Signed-off-by: xinyual <[email protected]>
@xinyual
verify
5ae0bcd 
Signed-off-by: xinyual <[email protected]>
@xinyual
undo string utils
2ba663a 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip dummy model
6c1259a 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip dummy model
afdf235 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip dummy model
5a928fe 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip dummy model
6018da8 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip dummy model
eb5f944 
Signed-off-by: xinyual <[email protected]>
@xinyual
skip dummy model
9bf24b4 
Signed-off-by: xinyual <[email protected]>
@xinyual
add inner load Model
cd3e513 
Signed-off-by: xinyual <[email protected]>
@xinyual
rename variable
597ab68 
Signed-off-by: xinyual <[email protected]>
@xinyual
add default for idf
e1c8b5e 
Signed-off-by: xinyual <[email protected]>
@xinyual
add ut for sparse encoding and tokenizer
0d14e8b
@xinyual
add close model
4c82bad 
Signed-off-by: xinyual <[email protected]>
@xinyual
change mock class
ced1323 
Signed-off-by: xinyual <[email protected]>
@xinyual
remove buffer for sparse encoding output
c651a4e 
Signed-off-by: xinyual <[email protected]>
@xinyual xinyual force-pushed the addTokenizerAndSparseEncoding branch from 880a4a0 to f98f52a Compare September 12, 2023 11:46
@xinyual xinyual force-pushed the addTokenizerAndSparseEncoding branch 2 times, most recently from a41690f to 4e3dc78 Compare September 26, 2023 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@xinyual