Skip to content

Add Whisper Speech-to-Text #1114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

Add Whisper Speech-to-Text #1114

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e822b07
Pass activation in conv layer
abheesht17 Jun 30, 2023
539e88c
Fixes
abheesht17 Jun 30, 2023
b39a96a
Fixes (1)
abheesht17 Jun 30, 2023
da6771b
Format
abheesht17 Jun 30, 2023
7046ae7
Add AudioToSpeechLM and corresponding preprocessor
abheesht17 Jul 2, 2023
6cbff40
Add end_token_id as a mem var for tok
abheesht17 Jul 2, 2023
2492f97
Minor edit
abheesht17 Jul 2, 2023
c75f4a9
Expt (1)
abheesht17 Jul 10, 2023
21c49cd
Support multilingual Whisper AtS
abheesht17 Jul 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Pass activation in conv layer
  • Loading branch information
@abheesht17
abheesht17 committed Jun 30, 2023
commit e822b071d839cd6cc82768cf130853783ff641ea
12 changes: 4 additions & 8 deletions keras_nlp/models/whisper/whisper_backbone.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,12 +135,10 @@ def __init__(
kernel_size=3,
strides=1,
padding="same",
activation=keras.activation.gelu(approximate=False),
name="encoder_token_embedding_conv_layer_1",
)
embedded_features = keras.activations.gelu(
encoder_conv_layer_1(encoder_feature_input),
approximate=False,
)
embedded_features = encoder_conv_layer_1(encoder_feature_input)

# For the second conv. layer, we cannot use `padding="same"` since
# that corresponds to a padding size of 1.5 (since stride is 2). Hence,
Expand All @@ -153,12 +151,10 @@ def __init__(
kernel_size=3,
strides=2,
padding="valid",
activation=keras.activation.gelu(approximate=False),
name="encoder_token_embedding_conv_layer_2",
)
embedded_features = keras.activations.gelu(
encoder_conv_layer_2(embedded_features),
approximate=False,
)
embedded_features = encoder_conv_layer_2(embedded_features)

# The position embedding layer for the encoder is a sinusoidal embedding
# layer: https://github.com/openai/whisper/blob/v20230124/whisper/model.py#L137.
Expand Down