-
Notifications
You must be signed in to change notification settings - Fork 356
mace-freeze merge #830
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
mace-freeze merge #830
Conversation
|
Fixed AttributeError: 'NoneType' object has no attribute 'layout' in the check gradients function, the pre-commit tests look fine on my machine, hopefully no more issues |
|
The latest addition is learning rate rescaling, which acts as a 'soft-freeze' mode. It assigns a low learning rate to certain parameters instead of freezing them completely. This can be used alongside freezing as a buffer (frozen par → soft-frozen par → active par), with multihead fine-tuning (for the deeper layers), or on its own. I'll add instructions to the README soon, along with some tests. |
|
*this update is only compatible with cueq 0.4.0, the earlier cueq versions with mace 0.3.13 do not handle None grads |
The addition of frozen transfer learning functionality (freezing layers/parameter tensors) for fine-tuning of foundation models. Please refer to the updates in README.md file for details, and cite https://arxiv.org/abs/2502.15582