Skip to content

Add Bloom Model #1382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Jan 15, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
438bf01
Add Bloom Model
abuelnasr0 Dec 27, 2023
bb281d9
Add Backbone test and some fixes
abuelnasr0 Dec 28, 2023
762f1a0
Add BloomBackbone to keras_nlp.models
abuelnasr0 Dec 28, 2023
a01fcd8
Fix a typo in layer naming
abuelnasr0 Jan 1, 2024
889f204
Remove self.built = True
abuelnasr0 Jan 1, 2024
7e0313f
Revert "Remove self.built = True"
abuelnasr0 Jan 1, 2024
6eba2fa
Add built=True to MLP layer
abuelnasr0 Jan 1, 2024
b77a22e
Add Checkpoint conversion script
abuelnasr0 Jan 1, 2024
a61267f
Change LayerNorm name
abuelnasr0 Jan 1, 2024
f64f532
Fix typo
abuelnasr0 Jan 1, 2024
103664a
Fix getting HF model output
abuelnasr0 Jan 1, 2024
52a1160
Add and to allclose function in checkpoint conversion script
abuelnasr0 Jan 1, 2024
8017f4e
Remove allclose check
abuelnasr0 Jan 1, 2024
d700931
Add doc for bloom
abuelnasr0 Jan 2, 2024
3461a3e
Write batch size instead of _
abuelnasr0 Jan 6, 2024
5964185
Rename out_dense to output_dense
abuelnasr0 Jan 6, 2024
7c95c10
Rename out_dense to output_dense
abuelnasr0 Jan 6, 2024
069f0d2
Format to 80 chars and remove unnecessery check
abuelnasr0 Jan 6, 2024
d2514c9
Remove exporting BloomDecoder
abuelnasr0 Jan 6, 2024
344c903
Add intermediate_dim Arg
abuelnasr0 Jan 6, 2024
b2076ff
Format the code
abuelnasr0 Jan 6, 2024
fd9c64c
Remove unnecessery comment
abuelnasr0 Jan 6, 2024
4a5a114
Use keras gelu
abuelnasr0 Jan 6, 2024
1136ca2
Remove MLP layer and implement it inside BloomDecoder
abuelnasr0 Jan 6, 2024
2d03d2c
Split q k v heads
abuelnasr0 Jan 6, 2024
2eeb5f4
Remove shapes comments
abuelnasr0 Jan 6, 2024
531b1ff
Revert "Split q k v heads"
abuelnasr0 Jan 6, 2024
786677b
Revert "Revert "Split q k v heads""
abuelnasr0 Jan 6, 2024
eccef98
Revert "Remove shapes comments"
abuelnasr0 Jan 6, 2024
ea3063d
Add bias axes
abuelnasr0 Jan 6, 2024
38826c4
Add bias axes to the correct axes
abuelnasr0 Jan 6, 2024
2ad1d17
Update conversion script for splitting q,k,v
abuelnasr0 Jan 6, 2024
b68300f
format the code
abuelnasr0 Jan 6, 2024
67f7198
Rename _dropout -> _dropout_layer
abuelnasr0 Jan 7, 2024
33ddca2
use clone initializer instead of paasing str name
abuelnasr0 Jan 7, 2024
e59e9e2
Serialize kernal & bais initializers
abuelnasr0 Jan 7, 2024
56e4492
Format the code
abuelnasr0 Jan 7, 2024
d89a207
Add alibi_bias_max to _build_alibi_tensor function
abuelnasr0 Jan 7, 2024
0268dd7
Format the code
abuelnasr0 Jan 7, 2024
0c0233f
Lowercase vairiable names
abuelnasr0 Jan 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Format the code
  • Loading branch information
abuelnasr0 committed Jan 7, 2024
commit 56e4492cc1957d2931763046e8e7fb0e2855fe3c
2 changes: 1 addition & 1 deletion keras_nlp/models/bloom/bloom_attention.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,7 +204,7 @@ def get_config(self):
),
"bias_initializer": keras.initializers.serialize(
self.bias_initializer
)
),
}
)
return config
2 changes: 1 addition & 1 deletion keras_nlp/models/bloom/bloom_decoder.py
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ def get_config(self):
),
"bias_initializer": keras.initializers.serialize(
self.bias_initializer
)
),
}
)
return config