Allow storing scripts in multiple projects

[nielsbauman]

This commit makes the ScriptService, the ScriptCache, and the stored
script APIs project-aware. By adding the project ID to the cache key in
the ScriptCache, we avoid data leakage between projects. We resolve the
project ID from the thread context inside ScriptService#compile instead of
passing the project ID explicitly, to avoid making a large number of changes.
Since scripts should always be compiled in a user context, resolving the
project ID this way should be fine.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[rjernst]

ScriptService#compile has a lot of usages
Follow-up PRs will update the remaining usages to pass the correct project ID.

Compilation should normally be in the context of a user request. Shouldn't it then get the project id from the thread context?

[nielsbauman]

Compilation should normally be in the context of a user request. Shouldn't it then get the project id from the thread context?

@rjernst, thanks for chiming in! We usually try to explicitly pass the project ID around and minimize the reliance on the thread context, as it can be difficult to track down where the project ID is supposed to be set in a call chain. Since there are a lot of usages all over the codebase here, I'm open to considering using the thread context. Can you think of any usages where compilation happens outside of the context of a user request? A perhaps counterintuitive example is cluster state update tasks; those are run without the thread context of the request. Do you happen to know if we do any script compilation in cluster state update tasks, for example? I can do some investigation if you don't already happen to know.

[rjernst]

Can you think of any usages where compilation happens outside of the context of a user request?

One I'm not sure of is async search. I assume the search happens in the user context, even after being forked to the async threadpool as a task, but if not then that would be one. Otherwise, scripts are a way for users to customize functionality, so it wouldn't make sense to compile a script outside a user request.

[nielsbauman]

Thanks, @rjernst. I pushed some changes to use the project resolver instead of explicitly passing a project ID to ScriptService#compile. I manually tested an async search with a stored script, and that seemed to work fine. I suggest that we stick to this approach until we run into any issues (which we probably won't based on your statement about scripts being compiled in user contexts). It would be great if you (or someone else from the Core/Infra team) could have a look, thanks!