feat: cuda support in transformers backend

[kno10]

The transformers backend does not appear to use the GPU.
I could not get any of the settings such as cuda: true or device: cuda to work. In the code, I could not see any of these options being used in the backend, either, yet.
Maybe copy the relevant lines from the diffusers backend?

LocalAI/backend/python/diffusers/backend_diffusers.py

Lines 223 to 233 in 9482acf

	if request.CUDA:
	self.pipe.to('cuda')
	# Assume directory from request.ModelFile.
	# Only if request.LoraAdapter it's not an absolute path
	if request.LoraAdapter and request.ModelFile != "" and not os.path.isabs(request.LoraAdapter) and request.LoraAdapter:
	# get base path of modelFile
	modelFileBase = os.path.dirname(request.ModelFile)
	# modify LoraAdapter to be relative to modelFileBase
	request.LoraAdapter = os.path.join(modelFileBase, request.LoraAdapter)
	device = "cpu" if not request.CUDA else "cuda"
	self.device = device

[mudler]

The transformers backend does not appear to use the GPU. I could not get any of the settings such as cuda: true or device: cuda to work. In the code, I could not see any of these options being used in the backend, either, yet. Maybe copy the relevant lines from the diffusers backend?

LocalAI/backend/python/diffusers/backend_diffusers.py

Lines 223 to 233 in 9482acf

	if request.CUDA:
	self.pipe.to('cuda')
	# Assume directory from request.ModelFile.
	# Only if request.LoraAdapter it's not an absolute path
	if request.LoraAdapter and request.ModelFile != "" and not os.path.isabs(request.LoraAdapter) and request.LoraAdapter:
	# get base path of modelFile
	modelFileBase = os.path.dirname(request.ModelFile)
	# modify LoraAdapter to be relative to modelFileBase
	request.LoraAdapter = os.path.join(modelFileBase, request.LoraAdapter)
	device = "cpu" if not request.CUDA else "cuda"
	self.device = device

good point - it was not hooked as I could not test it as well. please feel free to take a stab at it

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I see that the issue is related to the transformers backend not using the GPU. To enable the GPU usage for the transformers backend, you can modify the diffusers backend configuration file in the models directory and set the cuda parameter to true. Here's an example of a modified configuration file:

name: transformers_gpu
parameters:
  model: transformers
  backend: diffusers
  cuda: true
diffusers:
  pipeline_type: StableDiffusionPipeline
  cuda: true
  enable_parameters: "negative_prompt,num_inference_steps,clip_skip"
  scheduler_type: "k_dpmpp_sde"
  cfg_scale: 8
  clip_skip: 11

This configuration file sets the cuda parameter to true in the diffusers backend and enables the GPU usage for the transformers model. You can modify other parameters such as step, f16, or scheduler_type according to your needs.

Sources:

• https://localai.io/model-compatibility/diffusers/index.html
• https://localai.io/features/image-generation/index.html
• https://github.com/go-skynet/LocalAI/tree/master/backend/python/diffusers/diffusers.yml
• https://github.com/go-skynet/LocalAI/tree/master/docs/content/features/image-generation.md

[LeonSijiaLu]

I think this issue has been addressed https://github.com/mudler/LocalAI/blob/master/backend/python/transformers/transformers_server.py#L91, looks like we can close it, but please correct me if I missed something @mudler