Add Support for LoRA-XS

[RonanKMcGovern]

Feature request

Support LoRA-XS

Motivation

LoRA-XS allows for (sometimes) better fine-tuning results, with far fewer tunable parameters.

A and B matrices are initialised - and frozen! - based on an SVD.

An r matrix is placed between the A and B matrices, and this is all that needs tuning.

Your contribution

Repo here: https://github.com/MohammadrezaBanaei/LoRA-XS, I'll reach out to the authors.

[Rocketknight1]

cc @sayakpaul @BenjaminBossan for peft

[sayakpaul]

The official repo doesn't seem to have a lot of traction, which makes me question the adoption of this method in the community. I will let @BenjaminBossan @githubnemo to take a call.

[BenjaminBossan]

From the description, it sounds similar to MosLoRA (#2013, not merged). It would probably easy to implement as a LoRA variant (similar to how DoRA is implemented).

As a first step, I would suggest to ask directly on https://github.com/MohammadrezaBanaei/LoRA-XS whether they are interested in creating a PR to integrate LoRA-XS into PEFT. @RonanKMcGovern would you like to do that?