Skip to content

feat(metrics): add Glean metrics for passwordless signin#20227

Open
vbudhram wants to merge 1 commit intomainfrom
fxa-13019
Open

feat(metrics): add Glean metrics for passwordless signin#20227
vbudhram wants to merge 1 commit intomainfrom
fxa-13019

Conversation

@vbudhram
Copy link
Contributor

@vbudhram vbudhram commented Mar 20, 2026

Because

  • Passwordless signin/signup via OTP had no Glean telemetry, making it impossible to measure adoption, success rates, and error patterns
  • Backend login.complete and reg.complete events lacked a reason field to distinguish OTP completions from email/password or third-party auth

This pull request

  • Adds frontend Glean events for passwordless OTP flows: view, engage, submit, submitSuccess, error, and resendCode for both login and reg categories
  • Instruments SigninPasswordlessCode component with Glean events for the full lifecycle (view → engage → submit → success/error)
  • Adds reason extra key to backend login.complete and reg.complete events (values: email, otp, google, apple)
  • Defines all new events in fxa-ui-metrics.yaml and fxa-backend-metrics.yaml, regenerates TypeScript via glean-generate
  • Adds GleanEventsHelper Playwright utility that intercepts frontend Glean pings for functional test assertions

Issue

Closes: https://mozilla-hub.atlassian.net/browse/FXA-13019

Checklist

  • My commit is GPG signed
  • Tests pass locally (if applicable)
  • Documentation updated (if applicable)
  • RTL rendering verified (if UI changed)

Other Information

How to test:

  1. Navigate to an RP with force_passwordless=true
  2. Enter a new email → OTP code page should fire reg_otp_view
  3. Focus the code input → reg_otp_engage
  4. Submit valid code → reg_otp_submit then reg_otp_submit_success
  5. For existing passwordless account, events use login_otp_* prefix instead

@vbudhram vbudhram force-pushed the fxa-13019 branch 2 times, most recently from 7aba65c to 5f98d5d Compare March 20, 2026 17:05
@vbudhram vbudhram changed the title feat(metrics): add Glean metrics for passwordless signin (FXA-13019) feat(metrics): add Glean metrics for passwordless signin Mar 20, 2026
@vbudhram vbudhram force-pushed the fxa-13019 branch 3 times, most recently from 5611dd1 to ba24bfc Compare March 20, 2026 19:15
@vbudhram vbudhram marked this pull request as ready for review March 20, 2026 19:21
@vbudhram vbudhram requested a review from a team as a code owner March 20, 2026 19:21
Copilot AI review requested due to automatic review settings March 20, 2026 19:21
@vbudhram vbudhram requested a review from StaberindeZA March 20, 2026 19:22
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds end-to-end Glean telemetry for passwordless OTP registration/login, including new frontend OTP lifecycle events and a backend reason extra on login.complete / reg.complete so OTP vs password vs third-party completions can be distinguished.

Changes:

  • Define + generate new passwordless OTP frontend events for login and reg (view/engage/submit/success/error/resend).
  • Instrument the Settings passwordless code page to emit the new Glean events, with unit + functional test assertions.
  • Add a reason extra key to backend login.complete and reg.complete, and emit appropriate values from email, OTP, and third-party auth flows.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
packages/fxa-shared/metrics/glean/web/reg.ts Adds generated reg.otp_* event metric definitions.
packages/fxa-shared/metrics/glean/web/login.ts Adds generated login.otp_* event metric definitions.
packages/fxa-shared/metrics/glean/web/index.ts Adds eventsMap entries for passwordless OTP events used by the frontend wrapper.
packages/fxa-shared/metrics/glean/web/event.ts Generated import reordering.
packages/fxa-shared/metrics/glean/fxa-ui-metrics.yaml Declares new UI OTP events (login + reg) and extra_keys.
packages/fxa-shared/metrics/glean/fxa-backend-metrics.yaml Adds reason extra_key to backend login.complete / reg.complete.
packages/fxa-settings/src/pages/Signin/SigninPasswordlessCode/index.tsx Emits passwordless OTP lifecycle events from the code-entry page.
packages/fxa-settings/src/pages/Signin/SigninPasswordlessCode/index.test.tsx Adds unit tests asserting emitted passwordless OTP events.
packages/fxa-settings/src/lib/glean/index.ts Adds event-name → metric recording wiring for new OTP events.
packages/fxa-settings/src/components/FormVerifyCode/index.tsx Adds onEngageCb hook to emit OTP engage metric on first focus.
packages/fxa-auth-server/test/local/metrics/events.js Updates expected backend login.complete payload to include reason: 'email'.
packages/fxa-auth-server/lib/routes/utils/signup.js Emits reg.complete with reason: 'email' for standard email verification.
packages/fxa-auth-server/lib/routes/passwordless.ts Emits reg.complete / login.complete with reason: 'otp' for passwordless flows.
packages/fxa-auth-server/lib/routes/linked-accounts.ts Emits completion reasons for Google/Apple flows.
packages/fxa-auth-server/lib/metrics/glean/server_events.ts Adds reason to backend recordLoginComplete / recordRegComplete event extras.
packages/fxa-auth-server/lib/metrics/glean/index.ts Wires reason extra key into login_complete / reg_complete event creation.
packages/fxa-auth-server/lib/metrics/events.js Adds reason: 'email' when emitting login.complete on flow completion.
packages/functional-tests/tests/passwordless/signinPasswordless.spec.ts Adds functional assertions around OTP Glean event order + extras.
packages/functional-tests/lib/glean.ts New Playwright helper that intercepts frontend Glean pings for assertions.
packages/functional-tests/lib/fixtures/standard.ts Exposes gleanEventsHelper fixture for functional tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

* @param {string} utm_medium - The "medium" on which the user acted. For example, if the user clicked on a link in an email, then the value of this metric would be 'email'. The value has a max length of 128 characters with the alphanumeric characters, _ (underscore), forward slash (/), . (period), % (percentage sign), and - (hyphen) in the allowed set of characters..
* @param {string} utm_source - The source from where the user started. For example, if the user clicked on a link on the Mozilla accounts web site, this value could be 'fx-website'. The value has a max length of 128 characters with the alphanumeric characters, _ (underscore), forward slash (/), . (period), % (percentage sign), and - (hyphen) in the allowed set of characters..
* @param {string} utm_term - This metric is similar to the `utm.source`; it is used in the Firefox browser. For example, if the user started from about:welcome, then the value could be 'aboutwelcome-default-screen'. The value has a max length of 128 characters with the alphanumeric characters, _ (underscore), forward slash (/), . (period), % (percentage sign), and - (hyphen) in the allowed set of characters..
* @param {string} reason - Indicates how the user completed login. Values include "email" (traditional email/password), "otp" (passwordless OTP code)..
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The recordLoginComplete JSDoc for reason only mentions "email" and "otp", but linked-accounts.ts now emits "google" and "apple" reasons for third-party auth logins. Update the documented value set to match what the code can send.

Copilot uses AI. Check for mistakes.
Comment on lines +40 to +45
gleanEventsHelper.assertEventOrder([
'email_first_view',
'reg_otp_view',
'reg_otp_submit',
'reg_otp_submit_success',
]);
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These assertions can be flaky because Glean pings are sent asynchronously; assertEventOrder runs immediately after isLoggedIn() without waiting for the final ping(s) to be captured. Consider awaiting gleanEventsHelper.waitForEvent(...) for the last expected event (e.g. *_submit_success) before asserting order.

Copilot uses AI. Check for mistakes.
Comment on lines +639 to +649
gleanEventsHelper.assertEventOrder([
'reg_otp_view',
'reg_otp_submit',
'reg_otp_submit_frontend_error',
]);

const errorPings = gleanEventsHelper.getEventsByName(
'reg_otp_submit_frontend_error'
);
expect(errorPings.length).toBeGreaterThan(0);
expect(errorPings[0].extras.reason).toBe('invalid');
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same timing issue here: assertEventOrder() and subsequent getEventsByName()/extras assertions may run before the error ping is captured, leading to intermittent failures. Await gleanEventsHelper.waitForEvent('reg_otp_submit_frontend_error') (or similar) before asserting order and inspecting extras.reason.

Copilot uses AI. Check for mistakes.
Comment on lines +147 to +152
gleanOtp.error({ event: { reason: 'too many times' } });
setLocalizedErrorBannerMessage(
getLocalizedErrorMessage(ftlMsgResolver, error)
);
} else {
gleanOtp.error({ event: { reason: 'try again later' } });
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gleanOtp.error() maps to the *_otp_submit_frontend_error metric, but it's also being emitted for resend-code failures here. That will make resend failures look like submit failures in telemetry. Consider adding a dedicated resend error metric (e.g. *_otp_resend_frontend_error) or renaming/repurposing the existing error metric+docs to be a generic OTP frontend error for the whole page lifecycle.

Suggested change
gleanOtp.error({ event: { reason: 'too many times' } });
setLocalizedErrorBannerMessage(
getLocalizedErrorMessage(ftlMsgResolver, error)
);
} else {
gleanOtp.error({ event: { reason: 'try again later' } });
setLocalizedErrorBannerMessage(
getLocalizedErrorMessage(ftlMsgResolver, error)
);
} else {

Copilot uses AI. Check for mistakes.
extra_keys:
reason:
description: |
Indicates how the user completed login. Values include "email" (traditional email/password), "otp" (passwordless OTP code).
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

login.complete reason extra_key description only lists "email" and "otp", but auth-server now emits reason: 'google' | 'apple' for third-party auth logins as well. Please update this metric description (and ideally keep it in sync with the actual emitted value set) so downstream analysis/data-review expectations match reality.

Suggested change
Indicates how the user completed login. Values include "email" (traditional email/password), "otp" (passwordless OTP code).
Indicates how the user completed login. Values include "email" (traditional email/password), "otp" (passwordless OTP code),
"google" (third-party login via Google), and "apple" (third-party login via Apple). Additional values may be added over time
as new login methods are supported.

Copilot uses AI. Check for mistakes.
Comment on lines +2230 to +2237
reason: string;
}) {
const event = {
category: 'login',
name: 'complete',
extra: {
reason: String(reason),
},
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reason is typed as required and is always recorded into event.extra via String(reason). If any caller omits reason (or passes undefined), this will emit the literal string "undefined" into telemetry. Consider making reason optional here and only setting event.extra.reason when a non-empty value is provided (or defaulting it to an empty string earlier in the pipeline).

Suggested change
reason: string;
}) {
const event = {
category: 'login',
name: 'complete',
extra: {
reason: String(reason),
},
reason?: string;
}) {
const extra: Record<string, any> = {};
if (reason !== undefined && reason !== null && reason !== '') {
extra.reason = String(reason);
}
const event = {
category: 'login',
name: 'complete',
extra,

Copilot uses AI. Check for mistakes.
Comment on lines +221 to +223
complete: createEventFn('reg_complete', {
additionalMetrics: extraKeyReasonCb,
}),
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reg_complete/login_complete now forward reason via additionalMetrics: extraKeyReasonCb, but extraKeyReasonCb currently returns reason: metrics.reason which can be undefined when callers omit it. That ends up as String(undefined) in server_events.ts ("undefined" in telemetry). Recommend defaulting to '' in extraKeyReasonCb (or otherwise ensuring reason is always set) so these events never record an undefined reason.

Copilot uses AI. Check for mistakes.
Comment on lines +42 to +44
fetch(url, { method: 'POST', body: data, keepalive: true }).catch(
() => {}
);
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sendBeaconfetch shim uses default fetch options (mode: 'cors'). For cross-origin Glean endpoints (e.g. https://incoming.telemetry.mozilla.org/submit/...), this can trigger CORS/preflight behavior that sendBeacon normally avoids, and may prevent the POST from being sent/captured. Consider setting mode: 'no-cors' (and/or explicitly handling OPTIONS responses with appropriate Access-Control-Allow-* headers) to better emulate sendBeacon semantics.

Suggested change
fetch(url, { method: 'POST', body: data, keepalive: true }).catch(
() => {}
);
fetch(url, {
method: 'POST',
body: data,
keepalive: true,
mode: 'no-cors',
}).catch(() => {});

Copilot uses AI. Check for mistakes.
private pings: GleanPing[] = [];
private page: Page;
private started = false;
private readonly ROUTE_PATTERN = '**/submit/accounts-frontend*/**';
Copy link

Copilot AI Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ROUTE_PATTERN is hard-coded to **/submit/accounts-frontend*/**, but the frontend Glean applicationId default in this repo is accounts_frontend_dev (underscores) (see packages/fxa-settings/src/lib/config.ts / content-server config). If the upload path uses that applicationId verbatim, this route won't match and no events will be captured. Consider broadening the pattern (e.g. accounts*frontend*) or deriving it from the configured Glean applicationId used in the test environment.

Suggested change
private readonly ROUTE_PATTERN = '**/submit/accounts-frontend*/**';
// Match both hyphenated and underscored applicationIds, e.g. accounts-frontend* and accounts_frontend*
private readonly ROUTE_PATTERN = '**/submit/accounts*frontend*/**';

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants