[flytekit] Polish - Resources

[wild-endeavor]

Resources

This is a series of tickets to improve the flytekit authoring experience. If any changes are not possible to make in a backwards-compatible way, split it out into a separate ticket.

Rename argument

The argument ephemeral_storage should be renamed to disk. disk is succinct and gets the point across. Make it super clear if not already in user facing notes that the disk is ephemeral.

SHM

Should we add shm as an option to Resources? (i think the same issue that kept us from putting accelerator - the fact that resources are specified as both requests and limits - may be relevant here. Even if k8s doesn't allow it today, does it make sense to for an shm to have two different numbers here? Discuss more and probably cut a separate ticket for this (cc @granthamtaylor)

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[thomasjpfan]

I agree with changing ephemeral_storage to disk.

When adding shared memory with a pod template, it think it just sets the limit and it is not a request. I think most users just leaves it as default, which sets the limit as the total number of ram. Reference

A size limit can be specified for the default medium, which limits the capacity of the emptyDir volume. The storage is allocated from node ephemeral storage. If that is filled up from another source (for example, log files or image overlays), the emptyDir may run out of capacity before this limit. If no size is specified, memory backed volumes are sized to node allocatable memory.

This is what we do with the pytorch plugin: https://github.com/flyteorg/flytekit/blob/4e93e36843b8f13f06eb088e1a46232ad1fb225d/plugins/flytekit-kf-pytorch/flytekitplugins/kfpytorch/pod_template.py#L6. So we can put shm into Resource, but it can only be used with limit.

[JiangJiaWei1103]

#take