Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing metadata with underscores fail silently #878

Open
jbsilva opened this issue May 21, 2024 · 1 comment
Open

Writing metadata with underscores fail silently #878

jbsilva opened this issue May 21, 2024 · 1 comment

Comments

@jbsilva
Copy link

jbsilva commented May 21, 2024

Writing metadata containing _ is ignored without warnings.

More appropriate options would be:

  1. Allow underscores, as they are valid in AWS S3 (.getxattr() can't get them, as it performs a .replace("_", "-"))
  2. Warn or raise an exception
  3. Convert to -
file_bytes = b'My file...'
metadata = {"deviceId": 'LM1', 'device_serial': 'A1234', 'device-name': 'My device'}

fs= S3FileSystem()
fs.pipe_file("bucket/key", file_bytes, Metadata=metadata)

fs.metadata("bucket/key")
>> {'deviceid': 'LM1', 'device-name': 'My device'}

As you can see, the device_serial metadata was not written.

I can confirm this by looking in the S3 bucket or with boto3:

import boto3
s3_client = boto3.client('s3')
s3_client.head_object(Bucket="bucket", Key="key")["Metadata"]
>> {'deviceid': 'LM1', 'device-name': 'My device'}
@martindurant
Copy link
Member

Can you make PR to allow these keys, if you know how? I assume getxattr would need to be changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants