Why not conduct a requirement analysis before the plan?

[weicanie]

The practice of software engineering indicates that conducting a requirements analysis before planning is extremely necessary.
Especially in a toC agent application (this one), the user's requirements are often vague at the beginning. Therefore, clarifying the user's requirements before the agent makes a plan is necessary.
This is also a common pain point in agent applications. Initial errors accumulate in the agent loop and eventually cause everything to go haywire.

[matheusmaldaner]

Hi @weicanie, that is an excellent point, and the current system does try to achieve that in the get_orchestrator_system_message_planning() function with the planning prompt, particularly in the lines:

First consider the following:

    - is the user request missing information and can benefit from clarification? For instance, if the user asks "book a flight", the request is missing information about the destination, date and we should ask for clarification before proceeding. Do not ask to clarify more than once, after the first clarification, give a plan.

If you would like to contribute to make this a more explicit request (I noticed before that a few times it did not ask for clarifying information), I would be happy to review the PR. I believe a simple prompt fix there would work well, though I am pinging @husseinmozannar as he should have better insights.

[weicanie]

@matheusmaldaner Thank you for your response.I believe that this issue and improvement cannot be accomplished merely by fixing the prompt.

Firstly, prompts like "Is the user request missing information and can benefit from clarification?" are hard to be effective. Because even human find it difficult. And placing such a difficult question in a large prompt, LLMs are unlikely to pay sufficient attention to it and are more likely to skip it directly.

Secondly, this prompt assumes that the agent has sufficient knowledge to determine whether the current user request requires further clarification. Clearly, this assumption is fundamentally untenable.

[weicanie]

In my opinion, solving this problem in the "human-in-the-loop" manner is not only effective but also aligns with your goal of conducting human-centered exploration.

Imagine that when the plan is completed, the results directly linked to the user's input after all the steps are completed are displayed directly.
For example:

User input: I want to know which projects on GitHub are the most popular today.
Plan(mock):

• Step 1: Open the trending page of GitHub
• Step 2: Aggregate the popularity rankings of the projects and their information into a table for the user

Result (using the star rule as an example, I completed it using gemini-2.5-pro):

- **Situation:** The user desires to quickly grasp the current technological trends and popular projects in the open-source community, in order to discover new tools, frameworks, or learning directions. However, manually browsing and filtering through the information is inefficient. 

- **Task:** My task is to accurately and efficiently obtain the list of the top projects on GitHub for the current day, and then present the key information (such as project name, developer, number of stars, main language, and brief description) of these projects in a structured manner to the users. 

- **Action:**
1. I first carried out the first step of the plan by programmatically accessing the official GitHub trending page (`https://github.com/trending`).
2. Then, I parsed the returned page data, precisely extracting each project entry, and retrieved the core fields such as the project name, developers, today's Star count, project description, main programming language, and project link.
3. To make the information more intuitive, I consolidated and formatted these data into a clear table, and sorted it by popularity (today's Star count) from highest to lowest.
4. Finally, I presented this aggregated table containing all the key information to the user as the final result, ensuring that the user can easily obtain the required information at a glance. 

- **Result:** I successfully provided the users with a real-time and accurate ranking list of the most popular GitHub projects. Users no longer need to manually access and filter, but can directly see the list of projects that received the most attention from developers on that day. This table not only directly answers the core questions of the users, but also greatly facilitates their further exploration and research on the projects they are interested in by providing project summaries and links. It helps them efficiently keep up with the latest developments in the open-source community. 

From the user's perspective, I am a developer who is interested in LLM applications based on the agent paradigm. What I am more concerned about are agent-related projects.
After learning that this plan would provide me with a wide variety of projects, I modified my input:
Please provide me with the list of today's popular projects on GitHub, as well as the list of agent-related projects that are popular on GitHub today.

[weicanie]

I successfully solved this problem using this method, and I have fully realized that magentic-ui become more reliable and has a better user experience. If you also agree with this approach, I would be more than happy to submit a PR after providing sufficient test files.

Why do I think you would adopt this approach?

From a technical perspective:

• Make full use of the reasoning capabilities of LLM
• Address the issue of insufficient context in LLM
• Without disrupting the existing system and functions
• Easy to implement

From the user's perspective

• I have sufficient facts to revise the plan (instead of mentally rehearsing these steps, which can sometimes be quite painful)
• I can quickly input my ideas with confidence because I know everything is under my control
• Some of my vague requirements were quickly clarified, which made me feel that magentic-ui is really great

[matheusmaldaner]

Hi @weicanie, I completely agree with you, especially with the "placing such a difficult question in a large prompt, LLMs are unlikely to pay sufficient attention to it and are more likely to skip it directly" as I have experienced that before.

As for the STAR method/implementation, is that purely an orchestrator prompt change in _prompts.py? Or is there more logic changes on top of that? I am happy to take a look over the PR and share this with the team to see their thoughts!

Thank you so much as well for taking the time to make this contribution and for sharing these insights!