feat(ai) adding image tagging and translations #30287

wezell · 2024-10-08T15:58:14Z

This pull request introduces two new classes, AIVisionAPI and OpenAITranslationService, to enhance the AI capabilities in the dotCMS project. The AIVisionAPI class provides methods for tagging images and adding alt text, while the OpenAITranslationService class offers methods for translating content and handling translation parameters.

New AI Vision API:

dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.java: Added a new interface AIVisionAPI with methods for tagging images, adding alt text, and reading image tags and descriptions. ([dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.javaR1-R65](https://github.com/dotCMS/core/pull/30287/files#diff-57d9ce37bafde56eb885af1f975012f522647ef92554c2d97512da18bf322fb0R1-R65))

New AI Translation Service:

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java: Added a new class OpenAITranslationService that extends AbstractTranslationService and provides methods for translating strings and content, setting service parameters, and parsing AI responses. ([dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.javaR1-R328](https://github.com/dotCMS/core/pull/30287/files#diff-32212827a38e2210f004caac255ef701e833ac00077e894994a934c08c59b647R1-R328))

ref: #30284

dotCMS/src/main/java/com/dotcms/ai/workflow/OpenAITranslationActionlet.java

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java

dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.java

ref: #30284

dotcms-sonarqube · 2024-10-08T17:06:24Z

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

github-actions · 2024-11-08T02:00:10Z

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions · 2024-11-15T02:04:51Z

This PR was closed because it has been stalled with no activity.

ref: #30284

jdotcms · 2025-03-28T20:28:47Z

dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.java

+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+
+public interface AIVisionAPI {


jdotcms · 2025-03-28T20:29:47Z

dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.java

+
+    static final String AI_VISITON_TAG_AND_ALT_PROMPT_TEMPLATE = "AI_VISITON_TAG_AND_ALT_PROMPT_TEMPLATE";
+
+    static final Lazy<AIVisionAPI> instance = Lazy.of(com.dotcms.ai.api.OpenAIVisionAPIImpl::new);


I think future forward we have to have such as a factory or so, to be able to select the vendor and model depending on user parameters, but it is ok by now

jdotcms · 2025-03-28T20:29:55Z

dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.java

+    boolean tagImageIfNeeded(Contentlet contentlet);
+
+
+    boolean addAltTextIfNeeded(Contentlet contentlet);


jdotcms · 2025-03-28T20:30:13Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java

+import java.util.stream.Collectors;
+import org.apache.velocity.context.Context;
+
+public class OpenAITranslationService extends AbstractTranslationService {


jdotcms · 2025-03-28T20:30:52Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java

+
+public class OpenAITranslationService extends AbstractTranslationService {
+
+    static final String AI_TRANSLATION_SYSTEM_PROMPT = "AI_TRANSLATION_SYSTEM_PROMPT";


All of these seems to be a good candidate to be on super classes or interfaces

jdotcms · 2025-03-28T20:31:06Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java

+    static final String AI_TRANSLATION_TEMPERATURE ="AI_TRANSLATION_TEMPERATURE";
+    static final String AI_TRANSLATION_RESPONSE_FORMAT ="AI_TRANSLATION_RESPONSE_FORMAT";
+
+    static int MAX_LANGUAGE_VARIABLE_CONTEXT = 1000;


such as a config value?

jdotcms · 2025-03-28T20:35:03Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java

+        if (!translationKeysJSON.isEmpty()) {
+            systemContext.put("translationKeys", translationKeysJSON.toString());
+        }
+        String systemPrompt = Try.of(() -> VelocityUtil.eval(systemPromptTemplate, systemContext))


Prompting will be more common in the future, a Service to encapsulate their generation wouldnt be bad

jdotcms · 2025-03-28T20:35:56Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java

+    @Override
+    public Contentlet translateContent(Contentlet contentlet, Language targetLanguage, List<com.dotmarketing.portlets.structure.model.Field> oldFields, User user)
+            throws TranslationException {
+        Language sourceLang = APILocator.getLanguageAPI().getLanguage(contentlet.getLanguageId());


the method looks ok but it is a little bit longer, would be nice to make a refactor to split in into smaller methods

jdotcms · 2025-03-28T20:36:33Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAIVisionAPIImpl.java

+import java.util.stream.Collectors;
+import org.apache.velocity.context.Context;
+
+public class OpenAIVisionAPIImpl implements AIVisionAPI {


jdotcms · 2025-03-28T20:37:04Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAIVisionAPIImpl.java

+
+    static final ImageFilterExporter IMAGE_FILTER_EXPORTER = new ImageFilterExporter();
+
+    static final Cache<String, Tuple2<String, List<String>>> promptCache = Caffeine.newBuilder()


Yeah, the PromptService may handle cache or so

jdotcms · 2025-03-28T20:39:41Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAIVisionAPIImpl.java

+
+    private Optional<String> getSha256(File imageFile) {
+        try {
+            var md = MessageDigest.getInstance("SHA-256");


Encryptor.Hashing.sha256() does something similar to

jdotcms · 2025-03-28T20:41:07Z

dotCMS/src/main/java/com/dotcms/ai/api/OpenAIVisionAPIImpl.java

+            return Optional.empty();
+        }
+
+        final Context ctx = VelocityContextFactory.getMockContext(contentlet, APILocator.systemUser());


Not so fan of the Mock Context I would prefer that the factory realized if a request is tied in the current call and retrieve the full one, however I do not there are not too many fans of ThreadLocal approaches

jdotcms · 2025-03-28T20:42:15Z

dotCMS/src/main/java/com/dotcms/ai/listener/AIVisionInitializer.java

+
+    @Override
+    public void init() {
+        APILocator.getLocalSystemEventsAPI().subscribe(LISTENER);


jdotcms · 2025-03-28T20:42:31Z

dotCMS/src/main/java/com/dotcms/ai/listener/OpenAIImageTaggingContentListener.java

+import java.util.Optional;
+
+
+public class OpenAIImageTaggingContentListener implements ContentletListener<Contentlet> {


jdotcms · 2025-03-28T20:42:55Z

dotCMS/src/main/java/com/dotcms/ai/listener/OpenAIImageTaggingContentListener.java

+
+public class OpenAIImageTaggingContentListener implements ContentletListener<Contentlet> {
+
+    AIVisionAPI aiVisionAPI = AIVisionAPI.instance.get();


This should be via CDI or at least APILocator

jdotcms · 2025-03-28T20:43:43Z

dotCMS/src/main/java/com/dotcms/ai/util/AIUtil.java

+import java.util.Optional;
+import java.util.Properties;
+
+public class AIUtil {


jdotcms · 2025-03-28T20:44:30Z

dotCMS/src/main/java/com/dotcms/ai/util/AIUtil.java

+    private static final Properties properties;
+    static {
+        properties = new Properties();
+        try ( InputStream in = AIUtil.class.getResourceAsStream("/com/dotcms/ai/prompts/" + PROPERTY_FILE_NAME)){


This may be part of the PromptService / API

jdotcms · 2025-03-28T20:45:08Z

dotCMS/src/main/java/com/dotcms/ai/util/AIUtil.java

+    }
+
+    public static Map<String, Secret> getSecrets(String hostId) {
+        Host host = Try.of(() -> APILocator.getHostAPI().find(hostId, APILocator.systemUser(), true)).getOrNull();


I think secrets that contains private information should be destroyed at least at the end of the request

jdotcms · 2025-03-28T20:47:25Z

dotCMS/src/main/java/com/dotcms/ai/workflow/OpenAIVisionAutoTagActionlet.java

+import java.util.Map;
+import java.util.Optional;
+
+public class OpenAIVisionAutoTagActionlet extends WorkFlowActionlet {


jdotcms · 2025-03-28T20:47:44Z

dotCMS/src/main/java/com/dotcms/ai/workflow/OpenAIVisionAutoTagActionlet.java

+public class OpenAIVisionAutoTagActionlet extends WorkFlowActionlet {
+
+    private static final long serialVersionUID = 1L;
+    AIVisionAPI aiVisionAPI = AIVisionAPI.instance.get();


This should be in the CDI or APILocator

jdotcms · 2025-03-28T20:48:18Z

dotCMS/src/main/resources/com/dotcms/ai/prompts/default-vision-prompt.json

@@ -0,0 +1,21 @@
+{


We have to develop something to handle this thing in a standard way (dotCMS way) such as dotConnect or so, otherwise we will have N ways to do the same in a couple years

feat(ai) adding image tagging and translations

e685c94

ref: #30284

wezell linked an issue Oct 8, 2024 that may be closed by this pull request

Bring AI tagging,Alt descriptions and translations into core #30284

Closed

semgrep-code-dotcms-test bot reviewed Oct 8, 2024

View reviewed changes

dotCMS/src/main/java/com/dotcms/ai/workflow/OpenAITranslationActionlet.java Show resolved Hide resolved

semgrep-code-dotcms-test bot reviewed Oct 8, 2024

View reviewed changes

dotCMS/src/main/java/com/dotcms/ai/api/OpenAITranslationService.java Show resolved Hide resolved

semgrep-code-dotcms-test bot reviewed Oct 8, 2024

View reviewed changes

dotCMS/src/main/java/com/dotcms/ai/api/AIVisionAPI.java Show resolved Hide resolved

feat(ai) fixing sonarqube feedback

1ecfc87

ref: #30284

github-actions bot added the stale label Nov 8, 2024

github-actions bot closed this Nov 15, 2024

wezell reopened this Mar 20, 2025

Merge branch 'main' into issue-30284-pull-ai2-plugin-into-core

7c00c85

wezell marked this pull request as ready for review March 20, 2025 02:29

wezell added 2 commits March 21, 2025 10:05

Merge branch 'main' into issue-30284-pull-ai2-plugin-into-core

b5441f2

task(ai): updating ai-vision content listener

7537c73

ref: #30284

jcastro-dotcms approved these changes Mar 26, 2025

View reviewed changes

jdotcms reviewed Mar 28, 2025

View reviewed changes


		static final String AI_VISITON_TAG_AND_ALT_PROMPT_TEMPLATE = "AI_VISITON_TAG_AND_ALT_PROMPT_TEMPLATE";

		static final Lazy<AIVisionAPI> instance = Lazy.of(com.dotcms.ai.api.OpenAIVisionAPIImpl::new);

		boolean tagImageIfNeeded(Contentlet contentlet);


		boolean addAltTextIfNeeded(Contentlet contentlet);


		public class OpenAITranslationService extends AbstractTranslationService {

		static final String AI_TRANSLATION_SYSTEM_PROMPT = "AI_TRANSLATION_SYSTEM_PROMPT";


		static final ImageFilterExporter IMAGE_FILTER_EXPORTER = new ImageFilterExporter();

		static final Cache<String, Tuple2<String, List<String>>> promptCache = Caffeine.newBuilder()

		import java.util.Optional;


		public class OpenAIImageTaggingContentListener implements ContentletListener<Contentlet> {


		public class OpenAIImageTaggingContentListener implements ContentletListener<Contentlet> {

		AIVisionAPI aiVisionAPI = AIVisionAPI.instance.get();

feat(ai) adding image tagging and translations #30287

Are you sure you want to change the base?

feat(ai) adding image tagging and translations #30287

Uh oh!

Conversation

wezell commented Oct 8, 2024

New AI Vision API:

New AI Translation Service:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dotcms-sonarqube bot commented Oct 8, 2024

Quality Gate failed

Uh oh!

github-actions bot commented Nov 8, 2024

Uh oh!

github-actions bot commented Nov 15, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!