Skip to content

gguf-py: Refactor and allow reading/modifying existing GGUF files #3981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
Nov 11, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
b8c80df
gguf-py: Refactor and add file reading support
KerfuffleV2 Nov 7, 2023
8047aa1
Replay changes from #3871
KerfuffleV2 Nov 7, 2023
d7688dc
Various type annotation fixes.
KerfuffleV2 Nov 8, 2023
a6f5742
sort imports with isort (again)
cebtenzzre Nov 8, 2023
ce865b3
Fix missing return statement in add_tensor
KerfuffleV2 Nov 8, 2023
f364636
style cleanup with flake8
cebtenzzre Nov 8, 2023
f2292fc
fix NamedTuple and Enum usage
cebtenzzre Nov 8, 2023
fffdac3
Fix an issue with state init in GGUFReader
KerfuffleV2 Nov 8, 2023
b56ed66
Damagage is not a word.
KerfuffleV2 Nov 8, 2023
4a5cd69
Clean up gguf-py/examples/modify_gguf.py whitespace
KerfuffleV2 Nov 9, 2023
2af29ff
Update gguf-py/examples/modify_gguf.py formatting
KerfuffleV2 Nov 9, 2023
855486c
Update gguf-py/gguf/gguf_reader.py type hint
KerfuffleV2 Nov 9, 2023
2360aaa
Make examples executable, formatting changes
KerfuffleV2 Nov 9, 2023
8e250fe
Add more information to GGUFReader and examples comments
KerfuffleV2 Nov 9, 2023
0d0306e
Include a gguf Python package version bump
KerfuffleV2 Nov 9, 2023
cc58ad0
Merge branch 'master' into feat-gguf-py-read-refactor
KerfuffleV2 Nov 9, 2023
bca0962
Add convert-gguf-endian.py script
KerfuffleV2 Nov 9, 2023
233cb07
cleanup
cebtenzzre Nov 9, 2023
5738b2f
gguf-py : bump minor version
cebtenzzre Nov 9, 2023
52bdc7e
Reorganize scripts
KerfuffleV2 Nov 9, 2023
a04f048
Make GGUFReader endian detection less arbitrary
KerfuffleV2 Nov 9, 2023
bd241db
Add JSON dumping support to gguf-dump.py
KerfuffleV2 Nov 9, 2023
382f975
A few for gguf-dump.py cleanups
KerfuffleV2 Nov 10, 2023
7d3580d
Murder accidental tuple in gguf-py/scripts/gguf-dump.py
KerfuffleV2 Nov 10, 2023
5608cd8
cleanup
cebtenzzre Nov 10, 2023
795dc0f
constants : remove unneeded type annotations
cebtenzzre Nov 10, 2023
a21e9e7
fix python 3.8 compat
cebtenzzre Nov 10, 2023
eff662d
Set up gguf- scripts in pyproject.toml
KerfuffleV2 Nov 10, 2023
0b0e726
And include scripts/__init__.py, derp
KerfuffleV2 Nov 10, 2023
960f912
convert.py: We can't currently support Q8_0 on big endian.
KerfuffleV2 Nov 10, 2023
9ce51b6
gguf-py: SpecialVocab: Always try available sources for special token…
KerfuffleV2 Nov 10, 2023
f22b2f2
cleanup
cebtenzzre Nov 10, 2023
4814b4b
Promote add_X_token to GGUF metadata for BOS and EOS
KerfuffleV2 Nov 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
sort imports with isort (again)
  • Loading branch information
cebtenzzre committed Nov 8, 2023
commit a6f5742a53f4567a39fec6eee97aa654c1282841
2 changes: 1 addition & 1 deletion gguf-py/gguf/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
from .constants import *
from .gguf_writer import *
from .gguf_reader import *
from .gguf_writer import *
from .tensor_mapping import *
from .vocab import *
2 changes: 1 addition & 1 deletion gguf-py/gguf/constants.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from __future__ import annotations

import sys
from enum import Enum, IntEnum, auto, StrEnum
from enum import Enum, IntEnum, StrEnum, auto
from typing import Any, NamedTuple, Type

#
Expand Down
6 changes: 3 additions & 3 deletions gguf-py/gguf/gguf_reader.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

import os
from collections import OrderedDict
from typing import Any, TypeVar, NamedTuple, Dict, Type, Literal
from typing import Any, Dict, Literal, NamedTuple, Type, TypeVar

import numpy as np
import numpy.typing as npt
Expand All @@ -15,12 +15,12 @@
sys.path.insert(0, str(Path(__file__).parent.parent))

from gguf.constants import (
GGML_QUANT_SIZES,
GGUF_DEFAULT_ALIGNMENT,
GGUF_MAGIC,
GGUF_VERSION,
GGML_QUANT_SIZES,
GGMLQuantizationType,
GGUFValueType,
GGUFValueType
)

READER_SUPPORTED_VERSIONS = [2, GGUF_VERSION]
Expand Down
7 changes: 4 additions & 3 deletions gguf-py/gguf/gguf_writer.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@
import shutil
import struct
import tempfile
from io import BufferedWriter
from enum import Enum, auto
from typing import Any, IO, Sequence
from io import BufferedWriter
from typing import IO, Any, Sequence

import numpy as np

Expand All @@ -19,9 +19,10 @@
GGUFEndian,
GGUFValueType,
RopeScalingType,
TokenType,
TokenType
)


class WriterState(Enum):
EMPTY = auto()
HEADER = auto()
Expand Down
2 changes: 2 additions & 0 deletions gguf-py/gguf/tensor_mapping.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
from __future__ import annotations

from typing import Sequence

from .constants import MODEL_ARCH, MODEL_TENSOR, MODEL_TENSORS, TENSOR_NAMES


class TensorNameMap:
mappings_cfg: dict[MODEL_TENSOR, tuple[str, ...]] = {
# Token embeddings
Expand Down
1 change: 1 addition & 0 deletions gguf-py/gguf/vocab.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@

from .gguf_writer import GGUFWriter


class SpecialVocab:
merges: list[str]
special_token_ids: dict[str, int]
Expand Down