feat: 分块上传优化 #2813 #2816

zzdjx · 2024-12-03T08:02:23Z

No description provided.

...neric/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/GenericController.kt

.../common/common-mongo/src/main/kotlin/com/tencent/bkrepo/common/mongo/dao/AbstractMongoDao.kt

...main/kotlin/com/tencent/bkrepo/common/artifact/repository/core/AbstractArtifactRepository.kt

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

...ackend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/model/NodeAttribute.kt

...ce/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/node/impl/NodeDeleteSupport.kt

...ata-service/src/main/kotlin/com/tencent/bkrepo/common/metadata/dao/blocknode/BlockNodeDao.kt

...ice/src/main/kotlin/com/tencent/bkrepo/common/artifact/repository/core/ArtifactRepository.kt

...ce/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/node/impl/NodeDeleteSupport.kt

...metadata-service/src/main/kotlin/com/tencent/bkrepo/common/metadata/util/NodeDeleteHelper.kt

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

docs/apidoc/generic/SeparateBlock.md

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt

...ic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/artifact/GenericLocalRepository.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

...ata-service/src/main/kotlin/com/tencent/bkrepo/common/metadata/dao/blocknode/BlockNodeDao.kt

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

...ic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/artifact/GenericLocalRepository.kt

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

...etadata/metadata-api/src/main/kotlin/com/tencent/bkrepo/common/metadata/constant/Constant.kt

...main/kotlin/com/tencent/bkrepo/common/artifact/repository/core/AbstractArtifactRepository.kt

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

...end/generic/api-generic/src/main/kotlin/com/tencent/bkrepo/generic/pojo/SeparateBlockInfo.kt

...ic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/artifact/GenericLocalRepository.kt

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

cnlkl · 2025-01-10T11:57:19Z

...ice/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/BlockNodeService.kt

    /**
     * 删除旧分块，即删除非指定的nodeCurrentSha256的分块。
     * 如果未指定nodeCurrentSha256，则删除节点所有分块
+     * 如果指定uploadId，则删除该uploadId对应的分块，未指定则删除所有分块


未指定时指挥删除uploadId为null的分块，而不是所有分块

cnlkl · 2025-01-10T12:03:40Z

...ce/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/node/impl/NodeDeleteSupport.kt

+        if (deletedNum == 0L) return NodeDeleteResult(0L, 0L, deleteTime)
+
+        // 计算删除的文件大小
+        val deletedSize = nodeBaseService.aggregateComputeSize(criteria.apply {


这里不需要计算删除的文件大小

cnlkl · 2025-01-10T12:16:19Z

...ic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/artifact/GenericLocalRepository.kt

@@ -137,7 +145,9 @@ class GenericLocalRepository(
    private val replicaTaskClient: ReplicaTaskClient,
    private val clusterNodeClient: ClusterNodeClient,
    private val pipelineNodeService: PipelineNodeService,
-    private val ciPermissionManager: CIPermissionManager
+    private val ciPermissionManager: CIPermissionManager,
+    private val blockNodeServiceImpl: BlockNodeServiceImpl,


这里要依赖接口

cnlkl · 2025-01-10T12:18:28Z

...ic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/artifact/GenericLocalRepository.kt

+            val sha256 = getArtifactSha256()
+
+            val offset = context.request.getHeader(HEADER_OFFSET)?.toLongOrNull()
+            val expires = storageProperties.filesystem.cache.expireDuration


取错配置了

cnlkl · 2025-01-10T12:22:25Z

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

+
+            val node = nodeService.getNodeDetail(this)
+
+            val expires = getLongHeader(HEADER_EXPIRES).takeIf { it > 0 } ?: TRANSACTION_EXPIRES


这里不需要支持用户配置，支持服务内配置限制最大过期时间即可

cnlkl · 2025-01-10T12:24:39Z

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt

+        )
+
+        // 获取节点信息并验证节点是否存在
+        val node = ArtifactContextHolder.getNodeDetail(artifactInfo) ?: run {


此处为什么要验证节点是否存在

feat: 分块上传优化 TencentBlueKing#2813

8642550

zzdjx requested review from cnlkl, yaoxuwan and felixncheng as code owners December 3, 2024 08:02

zzdjx requested review from owenlxu and removed request for cnlkl, felixncheng and yaoxuwan December 3, 2024 08:02

fix: 分块上传优化-问题修复 TencentBlueKing#2813

5cbe04d

owenlxu reviewed Dec 4, 2024

View reviewed changes

...neric/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/GenericController.kt Outdated Show resolved Hide resolved

owenlxu reviewed Dec 4, 2024

View reviewed changes

.../common/common-mongo/src/main/kotlin/com/tencent/bkrepo/common/mongo/dao/AbstractMongoDao.kt Outdated Show resolved Hide resolved

owenlxu reviewed Dec 4, 2024

View reviewed changes

...main/kotlin/com/tencent/bkrepo/common/artifact/repository/core/AbstractArtifactRepository.kt Outdated Show resolved Hide resolved

owenlxu reviewed Dec 4, 2024

View reviewed changes

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt Outdated Show resolved Hide resolved

owenlxu reviewed Dec 4, 2024

View reviewed changes

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt Outdated Show resolved Hide resolved

zzdjx added 3 commits December 6, 2024 16:36

Merge remote-tracking branch 'upstream/master' into dev_kurt

613b9f8

fix: 分块上传优化-代码调整+单测 TencentBlueKing#2813

a80c07a

fix: 分块上传优化-代码调整 TencentBlueKing#2813

d65bc84

zzdjx requested review from yaoxuwan, cnlkl, zacYL and felixncheng December 9, 2024 07:52

yaoxuwan reviewed Dec 10, 2024

View reviewed changes

...ackend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/model/NodeAttribute.kt Outdated Show resolved Hide resolved

zzdjx added 2 commits December 10, 2024 15:01

Merge remote-tracking branch 'upstream/master' into dev_kurt

8009fcf

fix: NodeAttribute抽离公共放到metadata TencentBlueKing#2813

c21d488

yaoxuwan reviewed Dec 10, 2024

View reviewed changes

...ce/src/main/kotlin/com/tencent/bkrepo/common/metadata/service/node/impl/NodeDeleteSupport.kt Outdated Show resolved Hide resolved

zzdjx added 3 commits December 11, 2024 15:16

fix: 新增Generic下Block事件监听 TencentBlueKing#2813

318f65a

fix: 代码规范 TencentBlueKing#2813

ba35f7f

fix: 错误修复 TencentBlueKing#2813

89a7905

cnlkl requested changes Dec 11, 2024

View reviewed changes

zacYL reviewed Dec 12, 2024

View reviewed changes

docs/apidoc/generic/SeparateBlock.md Outdated Show resolved Hide resolved

...biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/controller/SeparateBlockController.kt Outdated Show resolved Hide resolved

fix: 代码规范 TencentBlueKing#2813

8e0f621

zzdjx added the for test label Dec 19, 2024

felixncheng previously approved these changes Dec 25, 2024

View reviewed changes

zacYL reviewed Dec 25, 2024

View reviewed changes

...ain/kotlin/com/tencent/bkrepo/common/metadata/service/blocknode/impl/BlockNodeServiceImpl.kt Outdated Show resolved Hide resolved

zacYL reviewed Dec 25, 2024

View reviewed changes

...ic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/artifact/GenericLocalRepository.kt Outdated Show resolved Hide resolved

zacYL reviewed Dec 25, 2024

View reviewed changes

...kend/generic/biz-generic/src/main/kotlin/com/tencent/bkrepo/generic/service/UploadService.kt Outdated Show resolved Hide resolved

fix: 代码格式及部分逻辑优化 TencentBlueKing#2813

11f5525

zzdjx dismissed felixncheng’s stale review via 11f5525 December 25, 2024 11:33

zzdjx added tested and removed for test tested labels Dec 26, 2024

fix: 修复isSeparateUpload uploadId null异常 TencentBlueKing#2813

b940e8c

zzdjx force-pushed the issue_2813 branch from 3fd0999 to b940e8c Compare January 2, 2025 10:35

cnlkl requested changes Jan 2, 2025

View reviewed changes

zzdjx added 8 commits January 6, 2025 11:31

feat: 分块上传并发场景下优化 TencentBlueKing#2813

1ca4460

fix: 代码优化 TencentBlueKing#2813

2804cbb

fix: test方法 TencentBlueKing#2813

020d69b

fix: 修复问题 TencentBlueKing#2813

71ac504

fix: 修复问题 TencentBlueKing#2813

b7c980d

fix: 文档更新 TencentBlueKing#2813

3ae6dac

fix: 增加部分log TencentBlueKing#2813

f33e49c

fix: 调试日志 TencentBlueKing#2813

637fac2

cnlkl self-requested a review January 6, 2025 11:42

zzdjx added 3 commits January 6, 2025 19:54

fix: 调试日志 TencentBlueKing#2813

5cbba61

fix: 问题修复 TencentBlueKing#2813

6469506

fix: 问题修复 TencentBlueKing#2813

28ba5d7

cnlkl requested changes Jan 8, 2025

View reviewed changes

zzdjx added 2 commits January 9, 2025 18:18

feat: blocks 新增过期逻辑 TencentBlueKing#2813

7b53435

feat: 过期时间改为配置项 TencentBlueKing#2813

c3fb6fa

cnlkl requested changes Jan 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: 分块上传优化 #2813 #2816

feat: 分块上传优化 #2813 #2816

zzdjx commented Dec 3, 2024

cnlkl Jan 10, 2025

cnlkl Jan 10, 2025

cnlkl Jan 10, 2025

cnlkl Jan 10, 2025

cnlkl Jan 10, 2025

cnlkl Jan 10, 2025


		val node = nodeService.getNodeDetail(this)

		val expires = getLongHeader(HEADER_EXPIRES).takeIf { it > 0 } ?: TRANSACTION_EXPIRES

feat: 分块上传优化 #2813 #2816

Are you sure you want to change the base?

feat: 分块上传优化 #2813 #2816

Conversation

zzdjx commented Dec 3, 2024

cnlkl Jan 10, 2025

Choose a reason for hiding this comment

cnlkl Jan 10, 2025

Choose a reason for hiding this comment

cnlkl Jan 10, 2025

Choose a reason for hiding this comment

cnlkl Jan 10, 2025

Choose a reason for hiding this comment

cnlkl Jan 10, 2025

Choose a reason for hiding this comment

cnlkl Jan 10, 2025

Choose a reason for hiding this comment