Skip to content

Reduce memory usage during Whisper inference #431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Feb 4, 2023
Merged

Reduce memory usage during Whisper inference #431

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
60eff46
ggml : add "scratch" buffer support
ggerganov Jan 19, 2023
0eea547
ggml : support for scratch ring-buffer
ggerganov Jan 21, 2023
1821057
ggml : bug fix in ggml_repeat()
ggerganov Jan 22, 2023
0ba91b5
ggml : error on scratch buffer overflow
ggerganov Jan 22, 2023
1a1dee4
whisper : use scratch buffers during inference (base model only)
ggerganov Jan 22, 2023
6cae05b
whisper : update memory usage for all models
ggerganov Jan 22, 2023
79148a2
whisper : fix encoder memory usage
ggerganov Jan 22, 2023
42d7dee
whisper : use whisper_context functions instead of macros
ggerganov Jan 25, 2023
4e0e252
whisper : fix FF + remove it from README
ggerganov Jan 25, 2023
d922aa4
ggml : reuse ggml_new_i32
ggerganov Jan 28, 2023
62205ae
ggml : refactor the scratch buffer storage
ggerganov Jan 28, 2023
01669ee
whisper : reorder scratch buffers in the decoder
ggerganov Jan 29, 2023
bdf21fa
main : add option to disable temp fallback
ggerganov Jan 29, 2023
6ed1344
Update README.md
ggerganov Jan 29, 2023
a26f3a7
js : update whisper.js
ggerganov Feb 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
whisper : use scratch buffers during inference (base model only)
  • Loading branch information
@ggerganov
ggerganov committed Feb 4, 2023
commit 1a1dee46ceab23581d1e44c48fdbd77526d7a8ba
2 changes: 1 addition & 1 deletion examples/main/main.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,7 @@ int main(int argc, char ** argv) {

for (int f = 0; f < (int) params.fname_inp.size(); ++f) {
const auto fname_inp = params.fname_inp[f];
const auto fname_outp = f < params.fname_outp.size() && !params.fname_outp[f].empty() ? params.fname_outp[f] : params.fname_inp[f];
const auto fname_outp = f < (int) params.fname_outp.size() && !params.fname_outp[f].empty() ? params.fname_outp[f] : params.fname_inp[f];

std::vector<float> pcmf32; // mono-channel F32 PCM
std::vector<std::vector<float>> pcmf32s; // stereo-channel F32 PCM
Expand Down
Loading