Require higher confidence before moving bugs out of Firefox :: General for users with editbugs

[gijsk]

In the past 3 weeks 3 of the bugs I filed in Fx :: General were moved out to other components. None of the moves were correct.

Discussing this with Marco, one potential solution could be requiring a higher confidence level from the bot that the bug belongs elsewhere when moving bugs filed by users with editbugs.

[suhaibmujahid]

@gijsk could you share these three bugs? I want to check their confidence levels when they were moved.

[suhaibmujahid]

I saw that you mentioned the bugs on Slack. Thank you!

Here are the bugs with the confidence values:

• https://bugzilla.mozilla.org/show_bug.cgi?id=1886999#c1: 87%
• https://bugzilla.mozilla.org/show_bug.cgi?id=1951642#c1: 93%
• https://bugzilla.mozilla.org/show_bug.cgi?id=1950892#c2: 82%

Their confidence values are relatively high, so I'm not sure that increasing the confidence threshold would help.

[gijsk]

Why are the confidence levels so high? Especially https://bugzilla.mozilla.org/show_bug.cgi?id=1886999#c1 moving to shopping makes absolutely 0 sense to me as a human - there is nothing relating to shopping in the bug, there are no comments that mention shopping or anything relating to shopping, no keywords, nothing.

[gijsk]

(also, the bug had happily lived in Fx::General for months - what prompted the bot to move it?)

[marco-c]

The model is not calibrated, so a high confidence doesn't necessarily mean a high probability. Maybe even if high, we should use an even higher one (or calibrate the model so the conficence is more like a probability).

[marco-c]

We can investigate those specific cases though, and figure out which characteristics of the bugs prompted the model to a specific label.

[gijsk]

Ran into this again in https://bugzilla.mozilla.org/show_bug.cgi?id=1981455#c3 . What could I do to help make progress here?

[marco-c]

@gijsk mozilla/bugbug#5183 might make this easier to resolve