[bounty] create a deduplication endpoint

[louis030195]

/bounty 150

definition of done:

• we have a deduplication endpoint that would help developer experience - making better agents / AI apps by removing the noise (see search pipe which already does it client side and it's quite bad UX because it blocks the UI thread and when there are tons of results it freeze for like 30+ seconds. using string similarity heuristics)
• should have a way to call it in the SDK like pipe.dedup(mydata) maybe taking the queryScreenpipe output as input? not sure, suggest good DX
• should use embedding model either using candle or onnx somehow (candle usually less pain but less models)
• most of the code in its own file imported then in server.rs
• should be as fast as possible and not destroying the user computer (use mac metal, or mkl/cuda if possible)

any thoughts before starting this?

[algora-pbc]

💎 $150 bounty • Screenpi.pe

Steps to solve:

1. Start working: Comment /attempt #1106 with your implementation plan
2. Submit work: Create a pull request including /claim #1106 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @b4s36t4	Jan 9, 2025, 6:01:05 AM	WIP

[b4s36t4]

/attempt #1106

Will put out the details on implementation here soon.

Algora profile	Completed bounties	Tech	Active attempts	Options
@b4s36t4	1 mediar-ai bounty + 3 bounties from 2 projects	TypeScript, Rust, JavaScript & more		Cancel attempt

[b4s36t4]

Embedding Model: jina-embeddings-v3, this does support multiple languages can be used with onnx as well.
SDK Action: I'm thinking to take the queryScreenpipe's input fields and let the backend do the query and return non-duplicated results.

Since SDK is in JS, passing data to rust might be costly in some cases, so good to do in rust itself.

New crates required:

• Candle (or) ORT - To Run Onnx Model.
• tokenizers - Tokenzie and embed
  please take a look and let me know if I need to make any changes or so.

cc: @louis030195