Skip to content

Conversation

@7radians
Copy link

The addition of frozen transfer learning functionality (freezing layers/parameter tensors) for fine-tuning of foundation models. Please refer to the updates in README.md file for details, and cite https://arxiv.org/abs/2502.15582

@7radians
Copy link
Author

Fixed AttributeError: 'NoneType' object has no attribute 'layout' in the check gradients function, the pre-commit tests look fine on my machine, hopefully no more issues

@7radians
Copy link
Author

7radians commented Apr 21, 2025

The latest addition is learning rate rescaling, which acts as a 'soft-freeze' mode. It assigns a low learning rate to certain parameters instead of freezing them completely. This can be used alongside freezing as a buffer (frozen par → soft-frozen par → active par), with multihead fine-tuning (for the deeper layers), or on its own. I'll add instructions to the README soon, along with some tests.

@7radians
Copy link
Author

7radians commented May 8, 2025

*this update is only compatible with cueq 0.4.0, the earlier cueq versions with mace 0.3.13 do not handle None grads

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant