Hani/ Testrail set all custom sub teste suites to functional #560

sv-hyacoub · 2025-04-25T13:08:04Z

Relevant Links

Bugzilla: 1962232

Description of Code / Doc Changes

Set all Sub Test Suites to "Functional" Only.

Process Changes Required

Mark the relevant boxes:

Adds a dependency (rerun pipenv install)
Changes the BasePage
Changes or creates a BOM/POM (name the object model): _
Changes CI flow
Changes scheduled Beta or DevEdition
Changes Git hooks or Github settings
Changes L10n harness

Screenshots or Explanations

If you need to explain your code, do it here.

Comments or Future Work

Do we need to start another PR soon to address something you saw while working on this?

Workflow Checklist

Please request reviewers
If this is an unblocker, please post in Slack.
If asked to address comments, please resolve conversations.
If asked to change code, please re-request review from the person who wanted changes.

Thank you!

sv-hyacoub · 2025-04-25T13:09:41Z

INFO:root:Found suites: [65, 67, 68, 73, 74, 85, 88, 102, 187, 498, 943, 1694, 1697, 1731, 1907, 1940, 1977, 1997, 1998, 2052, 2054, 2085, 2103, 2119, 2126, 2130, 2241, 2525, 2542, 3045, 5202, 5252, 5259, 5260, 5403, 5833, 6066, 6067, 6874, 18105, 18215, 22801, 23035, 24370, 29219, 42945, 43337, 43517, 49853, 52865, 53810, 54271, 54639, 54667, 55512, 55528, 55932, 56354, 56570, 56571, 56572, 56573, 56806, 57174, 57442, 58012, 58039, 58242, 58354, 59371, 60629, 60892, 60899, 61526, 61940, 61973, 61974, 62286, 62598, 63204, 63219, 64469, 64494, 64496, 64733]
INFO:root:Total cases fetched from suite 65: 95
INFO:root:Total cases fetched from suite 67: 6
INFO:root:Total cases fetched from suite 68: 67
INFO:root:Total cases fetched from suite 73: 192
INFO:root:Total cases fetched from suite 74: 41
INFO:root:Total cases fetched from suite 85: 77
INFO:root:Total cases fetched from suite 88: 18
INFO:root:Total cases fetched from suite 102: 94
INFO:root:Total cases fetched from suite 187: 5
INFO:root:Total cases fetched from suite 498: 6
INFO:root:Total cases fetched from suite 943: 48
INFO:root:Total cases fetched from suite 1694: 145
INFO:root:Total cases fetched from suite 1697: 86
INFO:root:Total cases fetched from suite 1731: 250
INFO:root:Total cases fetched from suite 1907: 56
INFO:root:Total cases fetched from suite 1940: 147
INFO:root:Total cases fetched from suite 1977: 35
INFO:root:Total cases fetched from suite 1997: 162
INFO:root:Total cases fetched from suite 1998: 15
INFO:root:Total cases fetched from suite 2052: 20
INFO:root:Total cases fetched from suite 2054: 250
INFO:root:Total cases fetched from suite 2085: 46
INFO:root:Total cases fetched from suite 2103: 86
INFO:root:Total cases fetched from suite 2119: 31
INFO:root:Total cases fetched from suite 2126: 8
INFO:root:Total cases fetched from suite 2130: 26
INFO:root:Total cases fetched from suite 2241: 83
INFO:root:Total cases fetched from suite 2525: 222
INFO:root:Total cases fetched from suite 2542: 160
INFO:root:Total cases fetched from suite 3045: 250
INFO:root:Total cases fetched from suite 5202: 3
INFO:root:Total cases fetched from suite 5252: 85
INFO:root:Total cases fetched from suite 5259: 41
INFO:root:Total cases fetched from suite 5260: 17
INFO:root:Total cases fetched from suite 5403: 250
INFO:root:Total cases fetched from suite 5833: 250
INFO:root:Total cases fetched from suite 6066: 80
INFO:root:Total cases fetched from suite 6067: 250
INFO:root:Total cases fetched from suite 6874: 47
INFO:root:Total cases fetched from suite 18105: 145
INFO:root:Total cases fetched from suite 18215: 250
INFO:root:Total cases fetched from suite 22801: 6
INFO:root:Total cases fetched from suite 23035: 155
INFO:root:Total cases fetched from suite 24370: 86
INFO:root:Total cases fetched from suite 29219: 221
INFO:root:Total cases fetched from suite 42945: 180
INFO:root:Total cases fetched from suite 43337: 102
INFO:root:Total cases fetched from suite 43517: 250
INFO:root:Total cases fetched from suite 49853: 8
INFO:root:Total cases fetched from suite 52865: 95
INFO:root:Total cases fetched from suite 53810: 98
INFO:root:Total cases fetched from suite 54271: 6
INFO:root:Total cases fetched from suite 54639: 18
INFO:root:Total cases fetched from suite 54667: 118
INFO:root:Total cases fetched from suite 55512: 13
INFO:root:Total cases fetched from suite 55528: 21
INFO:root:Total cases fetched from suite 55932: 65
INFO:root:Total cases fetched from suite 56354: 5
INFO:root:Total cases fetched from suite 56570: 62
INFO:root:Total cases fetched from suite 56571: 33
INFO:root:Total cases fetched from suite 56572: 74
INFO:root:Total cases fetched from suite 56573: 78
INFO:root:Total cases fetched from suite 56806: 25
INFO:root:Total cases fetched from suite 57174: 4
INFO:root:Total cases fetched from suite 57442: 12
INFO:root:Total cases fetched from suite 58012: 24
INFO:root:Total cases fetched from suite 58039: 117
INFO:root:Total cases fetched from suite 58242: 14
INFO:root:Total cases fetched from suite 58354: 91
INFO:root:Total cases fetched from suite 59371: 56
INFO:root:Total cases fetched from suite 60629: 27
INFO:root:Total cases fetched from suite 60892: 10
INFO:root:Total cases fetched from suite 60899: 3
INFO:root:Total cases fetched from suite 61526: 250
INFO:root:Total cases fetched from suite 61940: 53
INFO:root:Total cases fetched from suite 61973: 20
INFO:root:Total cases fetched from suite 61974: 31
INFO:root:Total cases fetched from suite 62286: 168
INFO:root:Total cases fetched from suite 62598: 18
INFO:root:Total cases fetched from suite 63204: 54
INFO:root:Total cases fetched from suite 63219: 39
INFO:root:Total cases fetched from suite 64469: 11
INFO:root:Total cases fetched from suite 64494: 8
INFO:root:Total cases fetched from suite 64496: 12
INFO:root:Total cases fetched from suite 64733: 8
INFO:root:Total test cases found for selected suites: 6894

I'm getting these testcases, but I'm not sure if we should update to all test suites in our project or not.

Tracy-Walker · 2025-04-28T12:12:56Z

Changing this in all the tests suites in Firefox desktop should be fine.

However, in every test suite the script hits the 250 cap, it will miss making the change in any test case beyond the first 250.

firefoxci-taskcluster · 2025-04-30T13:07:41Z

Uh oh! Looks like an error!

Client ID static/taskcluster/github does not have sufficient scopes and is missing the following scopes:

{
  "AnyOf": [
    "queue:rerun-task:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A/MHaTsS9ITGipYoYIHMo45Q",
    "queue:rerun-task-in-project:none",
    {
      "AllOf": [
        "queue:rerun-task",
        "assume:scheduler-id:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A"
      ]
    }
  ]
}

This request requires the client to satisfy the following scope expression:

{
  "AnyOf": [
    "queue:rerun-task:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A/MHaTsS9ITGipYoYIHMo45Q",
    "queue:rerun-task-in-project:none",
    {
      "AllOf": [
        "queue:rerun-task",
        "assume:scheduler-id:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A"
      ]
    }
  ]
}

method: rerunTask
errorCode: InsufficientScopes
statusCode: 403
time: 2025-04-30T13:07:41.207Z

firefoxci-taskcluster · 2025-04-30T13:07:57Z

Uh oh! Looks like an error!

Client ID static/taskcluster/github does not have sufficient scopes and is missing the following scopes:

{
  "AnyOf": [
    "queue:rerun-task:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A/MHaTsS9ITGipYoYIHMo45Q",
    "queue:rerun-task-in-project:none",
    {
      "AllOf": [
        "queue:rerun-task",
        "assume:scheduler-id:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A"
      ]
    }
  ]
}

This request requires the client to satisfy the following scope expression:

{
  "AnyOf": [
    "queue:rerun-task:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A/MHaTsS9ITGipYoYIHMo45Q",
    "queue:rerun-task-in-project:none",
    {
      "AllOf": [
        "queue:rerun-task",
        "assume:scheduler-id:mozilla-level-1/DBUJWwsfQVOrgACyR17g5A"
      ]
    }
  ]
}

method: rerunTask
errorCode: InsufficientScopes
statusCode: 403
time: 2025-04-30T13:07:57.316Z

ben-c-at-moz · 2025-04-30T19:09:08Z

Do we need to merge this at all, have we already run this script?

sv-hyacoub · 2025-05-05T13:59:45Z

Do we need to merge this at all, have we already run this script?

I don't think we need to merge it, I made the PR for a review. I didn't run the script.
Now I'm adding another verification to check that all test cases were updated correctly, but since there are 9688 tests locally its taking a lot of time(1h and counting), and I'm not sure how much time is normal or if we should update them all at once.

Testrail set all custom sub teste suites to functional

7d7973b

sv-hyacoub requested review from ben-c-at-moz, Tracy-Walker, soncuteanca and vsangereanMOZ April 25, 2025 13:31

adding pagination

acf1ed6

Merge branch 'main' into Hani/testrail_set_all_subs_to_functional

e8df445

Hani Yacoub added 2 commits May 6, 2025 15:29

add verifications

befb381

remove credentials

a64e20b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hani/ Testrail set all custom sub teste suites to functional #560

Hani/ Testrail set all custom sub teste suites to functional #560

sv-hyacoub commented Apr 25, 2025 •

edited

Loading

sv-hyacoub commented Apr 25, 2025

Tracy-Walker commented Apr 28, 2025

firefoxci-taskcluster bot commented Apr 30, 2025

firefoxci-taskcluster bot commented Apr 30, 2025

ben-c-at-moz commented Apr 30, 2025

sv-hyacoub commented May 5, 2025

Hani/ Testrail set all custom sub teste suites to functional #560

Are you sure you want to change the base?

Hani/ Testrail set all custom sub teste suites to functional #560

Conversation

sv-hyacoub commented Apr 25, 2025 • edited Loading

Relevant Links

Description of Code / Doc Changes

Process Changes Required

Screenshots or Explanations

Comments or Future Work

Workflow Checklist

sv-hyacoub commented Apr 25, 2025

Tracy-Walker commented Apr 28, 2025

firefoxci-taskcluster bot commented Apr 30, 2025

firefoxci-taskcluster bot commented Apr 30, 2025

ben-c-at-moz commented Apr 30, 2025

sv-hyacoub commented May 5, 2025

sv-hyacoub commented Apr 25, 2025 •

edited

Loading