Backend Clarification on set_tensor and get_tensor

[taronaeo]

The buffer set_tensor and get_tensor have the following function signatures

static void ggml_backend_zdnn_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size);

static void ggml_backend_zdnn_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size);

Do I have the following understanding correct?

1. .get_tensor copies the data from $ggml \ tensor \rightarrow backend \ tensor$
2. .set_tensor copies the data from $backend \ tensor \rightarrow ggml \ tensor$
3. data parameter is a contiguous buffer allocated by GGML and the actual data for the tensor is found via $data + offset$?
4. when writing the data back from the backend buffer, do we write into ggml_tensor or the data parameter?

I understand that this was briefly explained here but the wording was not very clear to me.

[slaren]

get_tensor and set_tensor copy data between a user buffer and a backend buffer. If the buffer is stored in the CPU and does not apply any transformations to the tensor data layout, the implementation is simply a memcpy:

llama.cpp/ggml/src/ggml-backend.cpp

Lines 1908 to 1918 in c8c4495

	static void ggml_backend_cpu_buffer_set_tensor(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, const void * data, size_t offset, size_t size) {
	memcpy((char *)tensor->data + offset, data, size);
	GGML_UNUSED(buffer);
	}
	static void ggml_backend_cpu_buffer_get_tensor(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, size_t offset, size_t size) {
	memcpy(data, (const char *)tensor->data + offset, size);
	GGML_UNUSED(buffer);
	}

If the backend does not store the tensor data in the standard ggml layout, you will need to apply any transformations in set_tensor and the inverse in get_tensor.

[slaren]

get_tensor is typically only called for obtaining the result of the computation, or to copy a tensor to a different backend.

[taronaeo]

Does this mean that immediately after every computation e.g., matmul, get_tensor will be called to (in my case), transform the data back and save into the buffer?

I'm not 100% certain yet but I don't think I see get_tensor being called after every computation and its throwing weird values because the data is not transformed back... I had to manually add the transform code within the compute routine to have it play nicely

[slaren]

No, users call get_tensor when they want to get the result of a graph evaluation, typically on the last tensor of the graph. For example, llama.cpp uses get_tensor to obtain the logits.

[taronaeo]

Oh my god, thats why the values are not being transformed back.

If;

1. init_tensor is used on every tensor to create things like zTensor and store it into tensor->extra,
2. set_tensor is used on every tensor to copy data into the backend i.e., into tensor->extra->ztensor,

where should we expect to transform values from tensor->extra->ztensor on every tensor back to normal values? I don't see any accompanying function that can help this other than to transform the value back within the compute routine, which likely means overhead 😅

[slaren]

Please head to https://github.com/ggml-org/ggml and take a look at the examples. Familiarize yourself with the ggml API before trying to implement a backend.