Skip to content

gguf-py: Improve GGUFReader read-only mode performance #10159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed

gguf-py: Improve GGUFReader read-only mode performance #10159

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
5ce2dbc
refactor gguf reader
Isotr0py Oct 31, 2024
bcef54e
improve performance
Isotr0py Oct 31, 2024
205676c
fix mode
Isotr0py Nov 4, 2024
dd320df
fix mode
Isotr0py Nov 4, 2024
1dc0215
optimize offsets calculation
Isotr0py Nov 4, 2024
a92c920
revert unnecessary change
Isotr0py Nov 4, 2024
ad6fd8d
revert unnecessary change
Isotr0py Nov 4, 2024
6a13722
code format
Isotr0py Nov 5, 2024
07ef1a8
make mode compatiable
Isotr0py Nov 5, 2024
810f06b
revert
Isotr0py Nov 5, 2024
94d814c
fix reader on linux
Isotr0py Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
code format
  • Loading branch information
@Isotr0py
Isotr0py committed Nov 5, 2024
commit 6a13722ca5c07c0b812bdc872c14949d10544234
5 changes: 2 additions & 3 deletions gguf-py/gguf/gguf_reader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

import logging
import os
import mmap
import struct
from collections import OrderedDict
from typing import Any, Literal, NamedTuple, TypeVar, Union
Expand Down Expand Up @@ -133,7 +132,7 @@ def __init__(self, path: os.PathLike[str] | str, mode: Literal['r', 'r+', 'c'] =
offs += self.alignment - padding
self.data_offset = offs
self._build_tensors(offs, tensors_fields)

def __del__(self) -> None:
self.data.close()

Expand Down Expand Up @@ -174,7 +173,7 @@ def _push_field(self, field: ReaderField, skip_sum: bool = False) -> int:
def _get_str(self, offset: int, return_size=False) -> tuple[npt.NDArray[np.uint64], npt.NDArray[np.uint8]]:
self.data.seek(offset)
slen = struct.unpack('<Q', self.data.read(8))
sdata = struct.unpack('<'+'B'*slen[0], self.data.read(1*slen[0]))
sdata = struct.unpack('<' + 'B' * slen[0], self.data.read(slen[0]))
output = (slen, sdata)
if return_size:
output += (8 + slen[0],)
Expand Down