sahi support in v0.2.0

[fcakyon]

What is the problem this feature will solve?

Related to this comment: #136 (comment)

I am wondering:

• What is your implementation plan?
• Do you want to include it as a tool or as a separate model?
• How can I help you with this?

What is the feature you are proposing to solve the problem?

Applying sliced inference for large images (as satellite images) using https://github.com/obss/sahi.

I believe it would be better to add it as a tool since it can work on top of many object detection instance segmentation models.

Or maybe we can add it as an inference-only model and the model type can be customized from the config file.

@hhaAndroid @PeterH0323

[fcakyon]

I am having this error when trying to download mmyolo config via python: open-mmlab/mim#170

[RangeKing]

I am having this error when trying to download mmyolo config via python: open-mmlab/mim#170

Hi @fcakyon, using mmyolo==0.1.2 will solve this problem.

[hhaAndroid]

@fcakyon Hi, We plan to integrate sahi in v0.1.3. The current plan is:

• First make a draft in mmyolo to see the generality
• If the generality is very strong, we can consider submitting PR directly to mmengine so that all openmmlab repo can be used directly

Can we write a draft version first and then discuss the next steps?

[fcakyon]

Can we write a draft version first and then discuss the next steps?

Sure 👍

[fcakyon]

@RangeKing when I run this code snippet:

from mmdet.apis import init_detector

config_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py'
model_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco_20220919_090739-b804c1ad.pth'

model = init_detector(
    config=config_path,
    checkpoint=model_path,
    device='cuda:0',
)

I get this error:

KeyError: 'YOLODetector is not in the model registry. Please check whether the value of `YOLODetector` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

env:
mmcv 2.0.0rc2 pypi_0 pypi
mmdet 3.0.0rc2 pypi_0 pypi
mmengine 0.3.0 pypi_0 pypi
mmyolo 0.1.2 pypi_0 pypi

Is there any way to use the inference API for mmyolo models?

[fcakyon]

I have fixed it by adding: register_all_modules()

[fcakyon]

I have another question, why is pycocotools a hard dependency for mmyolo? It is not necessary when doing only inference?

Opened a separate issue: #235

This problem is solved.

[fcakyon]

@hhaAndroid I am having another issue:

KeyError: 'LoadImageFromNDArray is not in the transform registry. Please check whether the value of `LoadImageFromNDArray` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

trying to perform inference using inference_detector(self.model, image) with numpy image.

[hhaAndroid]

@fcakyon Temporary solution:

from mmengine.registry import DefaultScope

with DefaultScope.overwrite_default_scope('mmdet'):
    self.predictor = init_detector(cfg, cfg.model_path)

[fcakyon]

@hhaAndroid now gives the following error during inference:

KeyError: 'YOLOv5KeepRatioResize is not in the transform registry. Please check whether the value of `YOLOv5KeepRatioResize` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

[fcakyon]

@hhaAndroid here is a list of issues I am facing:

Scenario1:

import numpy as np
from mmdet.apis import inference_detector, init_detector
from mmyolo.utils import register_all_modules
from mmengine.registry import DefaultScope

config_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py'
model_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco_20220919_090739-b804c1ad.pth'

register_all_modules()

model = init_detector(
    config=config_path,
    checkpoint=model_path,
    device='cuda:0',
)

prediction_result = inference_detector(model, np.random.randn(1, 3, 640, 640))

Error:

KeyError: 'LoadImageFromNDArray is not in the transform registry. Please check whether the value of `LoadImageFromNDArray` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

Scenario2:

import numpy as np
from mmdet.apis import inference_detector, init_detector
from mmyolo.utils import register_all_modules
from mmengine.registry import DefaultScope

config_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py'
model_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco_20220919_090739-b804c1ad.pth'

register_all_modules()

with DefaultScope.overwrite_default_scope("mmdet"):
    model = init_detector(
        config=config_path,
        checkpoint=model_path,
        device="cuda:0",
    )

prediction_result = inference_detector(model, np.random.randn(1, 3, 640, 640))

Error:

'YOLODetector is not in the model registry. Please check whether the value of `YOLODetector` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

Scenario3:

import numpy as np
from mmdet.apis import inference_detector, init_detector
from mmyolo.utils import register_all_modules
from mmengine.registry import DefaultScope

config_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py'
model_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco_20220919_090739-b804c1ad.pth'

with DefaultScope.overwrite_default_scope("mmdet"):
    register_all_modules()

    model = init_detector(
        config=config_path,
        checkpoint=model_path,
        device="cuda:0",
    )

prediction_result = inference_detector(model, np.random.randn(1, 3, 640, 640))

Error:

'LoadImageFromNDArray is not in the transform registry. Please check whether the value of `LoadImageFromNDArray` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

Scenario4:

import numpy as np
from mmdet.apis import inference_detector, init_detector
from mmyolo.utils import register_all_modules
from mmengine.registry import DefaultScope

config_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py'
model_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco_20220919_090739-b804c1ad.pth'

with DefaultScope.overwrite_default_scope("mmdet"):
    register_all_modules()

    model = init_detector(
        config=config_path,
        checkpoint=model_path,
        device="cuda:0",
    )

    prediction_result = inference_detector(model, np.random.randn(1, 3, 640, 640))

Error:

'LoadImageFromNDArray is not in the transform registry. Please check whether the value of `LoadImageFromNDArray` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

Scenario5:

import numpy as np
from mmdet.apis import inference_detector, init_detector
from mmyolo.utils import register_all_modules
from mmengine.registry import DefaultScope

config_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco.py'
model_path = 'yolov5_n-v61_syncbn_fast_8xb16-300e_coco_20220919_090739-b804c1ad.pth'

register_all_modules()

with DefaultScope.overwrite_default_scope("mmdet"):
    model = init_detector(
        config=config_path,
        checkpoint=model_path,
        device="cuda:0",
    )

    prediction_result = inference_detector(model, np.random.randn(1, 3, 640, 640))

Error:

'YOLODetector is not in the model registry. Please check whether the value of `YOLODetector` is correct or it was registered as expected. More details can be found at https://mmengine.readthedocs.io/en/latest/tutorials/config.html#import-custom-python-modules'

[hhaAndroid]

@fcakyon I'm very sorry. I didn't realize there were so many problems with cross-library calls now. It seems that overwrite_default_scope is not working and I have to rely on open-mmlab/mmdetection#9242 to fix it.

[fcakyon]

@fcakyon I'm very sorry. I didn't realize there were so many problems with cross-library calls now. It seems that overwrite_default_scope is not working and I have to rely on open-mmlab/mmdetection#9242 to fix it.

Is that update included in mmdet==v3.0.0rc3 or should I install mmdet v3 from source?

[liuyanyi]

For slice inference, there's already a huge_image_demo in mmrotate for large image slice inference, maybe we can refactor this feature in mmdet or mmcv to share this feature cross libraries.

Except for slice inference, I also expect there will be a slice evaluate just like coco evaluate in sahi. But evaluate in sahi only support coco, i hope that could support more dataset or more universal.

Slicing training is also a feature I was looking forward to. In DOTA dataset for rotate object detection, all image will be sliced to patches before training and testing, sliced datasets take up a lot of storage space. During paper reproduction, if some papers use a different slice size or strategy, the images need to be sliced again, its a huge burden for my hard drive.

Looking for your suggestion too.

[fcakyon]

For slice inference, there's already a huge_image_demo in mmrotate for large image slice inference, maybe we can refactor this feature in mmdet or mmcv to share this feature cross libraries.

yes that would be a great solution

[fcakyon]

Except for slice inference, I also expect there will be a slice evaluate just like coco evaluate in sahi. But evaluate in sahi only support coco, i hope that could support more dataset or more universal.

Actually sahi coco evaluate is similar to standard coco evaluation. It is not a sliced evaluation. We create coco formatted result.json after merging slice results.

[fcakyon]

Slicing training is also a feature I was looking forward to. In DOTA dataset for rotate object detection, all image will be sliced to patches before training and testing, sliced datasets take up a lot of storage space. During paper reproduction, if some papers use a different slice size or strategy, the images need to be sliced again, its a huge burden for my hard drive.

Creating a sliced dataset and then training on these images is the trivial solution. But if you want to save some disk space, these slicing operation can be performed during training on the fly as an augmentation with the help of dataloader maybe. For each loaded image, crop augmentation can be applied in a structured way (instead of randomly cropping) then these crop augmentations can be input into the model.

[liuyanyi]

Slicing training is also a feature I was looking forward to. In DOTA dataset for rotate object detection, all image will be sliced to patches before training and testing, sliced datasets take up a lot of storage space. During paper reproduction, if some papers use a different slice size or strategy, the images need to be sliced again, its a huge burden for my hard drive.

Creating a sliced dataset and then training on these images is the trivial solution. But if you want to save some disk space, these slicing operation can be performed during training on the fly as an augmentation with the help of dataloader maybe. For each loaded image, crop augmentation can be applied in a structured way (instead of randomly cropping) then these crop augmentations can be input into the model.

A slice dataset wrapper and pipeline may be a good solution, this can solve the preprocess, but how to handle the postprocess such as merge results? should the slicing during evaluation and inference be considered as type of test time augmentation

[fcakyon]

A slice dataset wrapper and pipeline may be a good solution, this can solve the preprocess, but how to handle the postprocess such as merge results? should the slicing during evaluation and inference be considered as type of test time augmentation

You are right, it would be better to handle it as test time augmentation so all slice results are merged back into the original image before performing image evaluation.

[hhaAndroid]

mmdet==v3.0.0rc3

mmdet==v3.0.0rc3

[hhaAndroid]

@liuyanyi Thank you very much for your feedback. I think you have a very good idea and it's something we really want to do. I'm not sure how much of a challenge it is for sahi to do this in combination with the OpenMMLab framework? Maybe we have to keep communicating and collaborating. Do we design a draft version of the proposal based on the requirements we have gathered so far? @fcakyon

[hhaAndroid]

@fcakyon Can you just provide a version that supports image_demo.py?

[fcakyon]

@hhaAndroid OK will work on it today.

[fcakyon]

I have made great progress, and will be able to open a pr with working large_image_demo.py using sahi this week 👍

[fcakyon]

@hhaAndroid @PeterH0323 created a pr adding large image demo using sahi: #284

Tested with yolov5_n-v61_syncbn_fast_8xb16-300e_coco and works fine.

What are your thoughts?

[hhaAndroid]

Thank you. I have replied

[fcakyon]

Thank you. I have replied

I have resolved your replies, can you check again in your free time? :)

[hhaAndroid]

Thank you. I have replied

I have resolved your replies, can you check again in your free time? :)

OK

[hhaAndroid]

@fcakyon Thank you very much, I made a few comments

[fcakyon]

@hhaAndroid thanks for the feedback. What is your opinion on adding large_image_inference api to mmdet?

[hhaAndroid]

@fcakyon My suggestion is to let users try it for a month and improve it before migrating to mmdet. What's your opinion?

[fcakyon]

@hhaAndroid sounds great 👍🏻

[thiagoribeirodamotta]

Was Sahi support added for online training? Is there any examples demonstrating how to use it?