Skip to content

Commit

Permalink
Fix behavior when Iterable is actually given as `split_to_filepatte…
Browse files Browse the repository at this point in the history
…rn`.

PiperOrigin-RevId: 599100213
  • Loading branch information
yotarok authored and SeqIO committed Jan 19, 2024
1 parent 9d4a4e4 commit 3bcb566
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion seqio/dataset_providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -695,10 +695,16 @@ def list_shards(self, split: str) -> Sequence[str]:
if isinstance(filepattern, str):
return _list_files(pattern=filepattern)

filepattern = list(filepattern)

if not any(glob.has_magic(f) for f in filepattern):
return filepattern
else:
return _list_files(pattern=filepattern)
assert isinstance(filepattern, Iterable)
ret = []
for f in filepattern:
ret.extend(_list_files(pattern=f))
return ret



Expand Down

0 comments on commit 3bcb566

Please sign in to comment.