Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File upload endpoint #205

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

File upload endpoint #205

wants to merge 4 commits into from

Conversation

bitbeckers
Copy link
Collaborator

The goal of this PR is to provide a basic IPFS upload endpoint to self-reported evaluations on hypercerts data. Primarily, it should allow impact creators to provide updates on the data represented withing an hypercert, like evaluation reports or proof of work/attendance/etc..

  • Adds upload endpoint gates with multen middleware
  • Cleans up types used in libs
  • Increased test coverage threshold based on updated test suite

Adds an upload endpoint to our API. This has been implemented using
multer as a middleware for file validations and restrictions on file
size. Basic happy/unhappy flow tests pass.
Increases coverage threshold based on updated test suite
Updates the types used in lib/users to match updated API types
Generated updated Swagger
@bitbeckers bitbeckers added the enhancement New feature or request label Jan 3, 2025
@bitbeckers bitbeckers requested a review from pheuberger January 3, 2025 01:13
@bitbeckers bitbeckers self-assigned this Jan 3, 2025
Copy link

github-actions bot commented Jan 3, 2025

Coverage Report

Status Category Percentage Covered / Total
🟢 Lines 18.64% (🎯 17%) 946 / 5073
🟢 Statements 18.64% (🎯 17%) 946 / 5073
🟢 Functions 56.62% (🎯 54%) 47 / 83
🟢 Branches 62.16% (🎯 60%) 161 / 259
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/index.ts 0% 0% 0% 0% 1-61
src/controllers/AllowListController.ts 90% 90% 100% 90% 107-114
src/controllers/MetadataController.ts 41.37% 72.72% 50% 41.37% 61-73, 116-180, 214-228, 237-244, 261-308
src/controllers/UploadController.ts 90% 90% 100% 90% 75-76, 153-159
src/lib/allowlists/isParsableToMerkleTree.ts 100% 100% 100% 100%
src/lib/allowlists/parseMerkleTree.ts 100% 100% 100% 100%
src/lib/users/EOAUpsertStrategy.ts 0% 0% 0% 0% 1-74
src/lib/users/MultisigUpsertStrategy.ts 0% 0% 0% 0% 1-81
src/lib/users/UserUpsertStrategy.ts 0% 0% 0% 0% 1-23
src/lib/users/errors.ts 0% 100% 100% 0% 3-11
src/middleware/upload.ts 84.09% 90% 50% 84.09% 55-61
src/utils/validateRemoteAllowList.ts 5.4% 100% 0% 5.4% 6-43
Generated in workflow #10 for commit 73c2e4b by the Vitest Coverage Report Action

Add validations on file type and size
To gradually clean up the repo, we agreed on restructuring helpers from
utils to more cleanly scoped libs. Additionally, added some tests
Copy link

@pheuberger pheuberger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some preliminary feedback on the controller. I haven't reviewed the tests yet and the later commits, but wanted to share already.

Tags,
UploadedFiles,
} from "tsoa";
import { StorageService } from "../services/StorageService.js";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking suggestion:

What are your thoughts on organizing imports by separating them into distinct groups with a newline in between?

Suggested order:

Global imports (e.g., libraries)
<newline>
Imports from files outside the current folder
<newline>
Imports from files within the same folder

* POST /v1/upload
* Content-Type: multipart/form-data
*
* files: [File, File, ...]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that really a correct example request? If I understand multipart form data correctly, shouldn't it be something like:

POST /v1/upload HTTP/1.1
Content-Type: multipart/form-data; boundary=SomeBoundaryXYZ

--SomeBoundaryXYZ
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain

<binary data of file1.txt>
--SomeBoundaryXYZ
Content-Disposition: form-data; name="files"; filename="image1.png"
Content-Type: image/png

<binary data of image1.png>
--SomeBoundaryXYZ--

}

const blobs = files?.map(
(file) => new Blob([file.buffer], { type: file.mimetype }),

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does Blob come from? Is that a native JS type?

fileName: file.originalname,
};
} catch (error) {
throw {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to create a custom Error here instead of throwing a JavaScript object.

);

const successful = uploadResults
.filter(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These next lines are really difficult to read and understand due to the generic, extra brackets and nesting. Would it make sense to extract the type narrowing function out into the module scope and use here?

.map((result) => result.value);

const failed = uploadResults
.filter(

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the above albeit much less difficult to understand.

if (failed.length > 0) {
this.setStatus(failed.length === uploadResults.length ? 500 : 207);
return {
success: false,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

success as boolean doesn't really cut it. How about a string field like this:

{
  uploaded: "all" | "some" | "none"
...
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants