feat(feedback): API endpoint for AI feedback summary #93134

vishnupsatish · 2025-06-09T18:29:37Z

Linear ticket: https://linear.app/getsentry/issue/REPLAY-292/api-endpoint-to-generate-summary-for-feedbacks

With tests

Includes changes from feat(feedback): create feedback usecase to generate AI summary #92932 (adding the feedbacksummaries usecase)
Q: Should the feature flag be added in this PR?
Todo: add the feedbacksummaries usecase with its respective model in getsentry options
--
PR to add feedbacksummaries LLM usecase: https://github.com/getsentry/getsentry/pull/17647
PRs to add feature flag: feat(feedback): Add temporary feature flag for AI user feedback summaries #93162 and https://github.com/getsentry/sentry-options-automator/pull/4115

…d uses create_feedback

codecov · 2025-06-09T18:34:11Z

Codecov Report

Attention: Patch coverage is 93.98148% with 13 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/feedback/usecases/feedback_summaries.py	69.56%	7 Missing ⚠️
...try/api/endpoints/organization_feedback_summary.py	88.46%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #93134      +/-   ##
==========================================
- Coverage   88.00%   87.95%   -0.05%     
==========================================
  Files       10260    10300      +40     
  Lines      591960   592519     +559     
  Branches    22978    22789     -189     
==========================================
+ Hits       520939   521138     +199     
- Misses      70531    70951     +420     
+ Partials      490      430      -60

src/sentry/api/endpoints/organization_feedback_summary.py

src/sentry/features/temporary.py

vishnupsatish · 2025-06-09T18:44:03Z

tests/sentry/api/endpoints/test_organization_feedback_summary.py

+    # is this a bad use of patch?
+    @patch(
+        "sentry.api.endpoints.organization_feedback_summary.MAX_FEEDBACKS_TO_SUMMARIZE_CHARS",
+        1000,
+    )


Bad use of patch?

src/sentry/api/endpoints/organization_feedback_summary.py

…format

vishnupsatish · 2025-06-10T21:22:49Z

are we waiting for a review on this from someone else?

michellewzhang · 2025-06-10T21:25:42Z

are we waiting for a review on this from someone else?

let's get @cmanallen to look at it

…perly??

vishnupsatish · 2025-06-10T23:13:17Z

Some tests in test_create_feedback are failing because the patch that was applied in test_organization_feedback_summary.py for some reason seem to not be cleaned up properly, so the monkeypatch that's applied in test_create_feedback doesn't work and it still uses the summary generation response.

tests/sentry/api/endpoints/test_organization_feedback_summary.py

cmanallen · 2025-06-11T14:54:21Z

src/sentry/feedback/usecases/feedback_summaries.py

+def make_input_prompt(
+    feedbacks,
+):
+    feedbacks_string = "\n".join(f"- {msg}" for msg in feedbacks)
+    return f"""Task:
+Instructions: You are an AI assistant that analyzes customer feedback.
+Create a summary based on the user feedbacks that is at most three sentences, and complete the sentence "Users say...". Be concise, but specific in the summary.
+
+User Feedbacks:
+
+{feedbacks_string}
+
+Output Format:
+
+Summary: <1-3 sentence summary>
+"""


Be aware. The AI team wants everything prompt related in Seer. I think its low risk to leave this here for now. But a future refactor should be considered where you proxy Seer.

cmanallen · 2025-06-11T14:56:15Z

src/sentry/feedback/usecases/feedback_summaries.py

+"""
+
+
+SUMMARY_REGEX = re.compile(r"Summary:\s*(.*)", re.DOTALL)


This is fine for now. But we can get structured output. There's many examples in Seer. Here's one I recently wrote: https://github.com/getsentry/seer/pull/2741/files#diff-798e03ed205a63e1198b21d0837bef5af0862f83704afafb084d68cff456ee4eR220-R251

src/sentry/feedback/usecases/feedback_summaries.py

src/sentry/api/endpoints/organization_feedback_summary.py

vishnupsatish · 2025-06-11T19:00:57Z

Updated this PR based on the reviews, currently waiting for tests to pass, is there anything else we need to do for this?

sentry-io · 2025-06-17T17:22:52Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ ValueError: Failed to parse AI feedback summary response /api/0/organizations/{organization_id_or_slug}/... View Issue

_{Did you find this useful? React with a 👍 or 👎}

Linear ticket: https://linear.app/getsentry/issue/REPLAY-292/api-endpoint-to-generate-summary-for-feedbacks With tests - Includes changes from #92932 (adding the `feedbacksummaries` usecase) - Q: Should the feature flag be added in this PR? - Todo: add the `feedbacksummaries` usecase with its respective model in getsentry options -- - PR to add feedbacksummaries LLM usecase: getsentry/getsentry#17647 - PRs to add feature flag: #93162 and getsentry/sentry-options-automator#4115

vishnupsatish added 10 commits June 5, 2025 09:10

feedback usecase to generate summary and parse it, and testing it

1af771f

update tests and some config

a15f6f7

update based on review - rename, strip consecutive whitespaces, etc.

9ba5ee5

add max tokens and add comment back

3b80ac2

create endpoint, very rough and no tests

937f1bf

create test file using cursor, need to change so it actually works an…

860cdb1

…d uses create_feedback

trying out tests and creating user feedbacks using kafka

376de16

write endpoint test that actually works

019c174

more tests for the API endpoint

0a3360f

format

aba030a

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jun 9, 2025

github-advanced-security bot found potential problems Jun 9, 2025

View reviewed changes

src/sentry/api/endpoints/organization_feedback_summary.py Fixed Show resolved Hide resolved

vishnupsatish added 3 commits June 9, 2025 11:35

fix docstring for endpoint

1956230

update comments

b16637f

remove custom exception for parse, fix failing test

b34742f

vishnupsatish commented Jun 9, 2025

View reviewed changes

src/sentry/features/temporary.py Outdated Show resolved Hide resolved

format tests

7368e0f

vishnupsatish commented Jun 9, 2025

View reviewed changes

vishnupsatish marked this pull request as ready for review June 9, 2025 18:44

vishnupsatish requested review from a team as code owners June 9, 2025 18:44

vercel bot deployed to Preview June 9, 2025 18:45 View deployment

vishnupsatish commented Jun 9, 2025

View reviewed changes

src/sentry/api/endpoints/organization_feedback_summary.py Outdated Show resolved Hide resolved

github-advanced-security bot found potential problems Jun 9, 2025

View reviewed changes

src/sentry/api/endpoints/organization_feedback_summary.py Fixed Show resolved Hide resolved

vishnupsatish changed the title ~~feat(feedback): API endpoint for AI summary of feedbacks~~ feat(feedback): API endpoint for AI feedback summary Jun 9, 2025

remove default=True for user-feedback-ui

c20e77a

vercel bot deployed to Preview June 9, 2025 19:21 View deployment

Merge branch 'master' into vishnupsatish/uf-summary-api-endpoint

4a1b805

vercel bot deployed to Preview June 10, 2025 18:57 View deployment

update url

6a9430d

vercel bot deployed to Preview June 10, 2025 19:07 View deployment

gate behind feature flag

faf5701

vercel bot deployed to Preview June 10, 2025 20:36 View deployment

test for nonexistent feature flag, test for different project qparam …

40dd6cc

…format

vercel bot deployed to Preview June 10, 2025 21:11 View deployment

michellewzhang requested a review from cmanallen June 10, 2025 21:25

merge master, why is a test_create_feedback test failing?

c6e3051

vercel bot deployed to Preview June 10, 2025 21:45 View deployment

use monkeypatch instead of patch, but it is still not cleaning up pro…

d630eef

…perly??

vercel bot deployed to Preview June 10, 2025 22:57 View deployment

revert to how it was before as monkeypatch isnt fixing the issue

5cda5e8

vercel bot deployed to Preview June 10, 2025 23:35 View deployment

aliu39 reviewed Jun 10, 2025

View reviewed changes

tests/sentry/api/endpoints/test_organization_feedback_summary.py Outdated Show resolved Hide resolved

cmanallen reviewed Jun 11, 2025

View reviewed changes

update based on review, mock generate_summary instead of openai

43ce9a4

vercel bot deployed to Preview June 11, 2025 19:01 View deployment

update url

ee740d1

vercel bot deployed to Preview June 11, 2025 22:13 View deployment

aliu39 approved these changes Jun 11, 2025

View reviewed changes

vishnupsatish merged commit 50c438b into master Jun 12, 2025
61 checks passed

vishnupsatish deleted the vishnupsatish/uf-summary-api-endpoint branch June 12, 2025 16:51

vishnupsatish mentioned this pull request Jun 12, 2025

feat(feedback): create feedback usecase to generate AI summary #92932

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(feedback): API endpoint for AI feedback summary #93134

feat(feedback): API endpoint for AI feedback summary #93134

vishnupsatish commented Jun 9, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jun 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

vishnupsatish Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

vishnupsatish commented Jun 10, 2025 •

edited

Loading

Uh oh!

michellewzhang commented Jun 10, 2025

Uh oh!

vishnupsatish commented Jun 10, 2025

Uh oh!

Uh oh!

cmanallen Jun 11, 2025

Uh oh!

cmanallen Jun 11, 2025

Uh oh!

Uh oh!

Uh oh!

vishnupsatish commented Jun 11, 2025

Uh oh!

Uh oh!

sentry-io bot commented Jun 17, 2025

Uh oh!

Uh oh!

		"""


		SUMMARY_REGEX = re.compile(r"Summary:\s(.)", re.DOTALL)

Uh oh!

feat(feedback): API endpoint for AI feedback summary #93134

feat(feedback): API endpoint for AI feedback summary #93134

Conversation

vishnupsatish commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

vishnupsatish Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vishnupsatish commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michellewzhang commented Jun 10, 2025

Uh oh!

vishnupsatish commented Jun 10, 2025

Uh oh!

Uh oh!

cmanallen Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

cmanallen Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vishnupsatish commented Jun 11, 2025

Uh oh!

Uh oh!

sentry-io bot commented Jun 17, 2025

Suspect Issues

Uh oh!

Uh oh!

vishnupsatish commented Jun 9, 2025 •

edited

Loading

codecov bot commented Jun 9, 2025 •

edited

Loading

vishnupsatish commented Jun 10, 2025 •

edited

Loading