Skip to content

Commit 132b530

Browse files
committed
Fix shard computation in NoShuffleBeamWriter with overlapping split
names
1 parent b5effb6 commit 132b530

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

tensorflow_datasets/core/writer.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -817,7 +817,7 @@ def finalize(self) -> tuple[list[int], int]:
817817
# We don't know the number of shards, the length of each shard, nor the
818818
# total size, so we compute them here.
819819
prefix = epath.Path(self._filename_template.filepath_prefix())
820-
shards = self._filename_template.data_dir.glob(f"{prefix.name}*")
820+
shards = self._filename_template.data_dir.glob(f"{prefix.name}.*")
821821

822822
def _get_length_and_size(shard: epath.Path) -> tuple[epath.Path, int, int]:
823823
length = self._file_adapter.num_examples(shard)

0 commit comments

Comments
 (0)