feat: add get_element_at(x, y) tool under --experimentalVision

[medyas]

Closes #268.

Adds a coordinate-driven element-inspection primitive that pairs with take_screenshot + a vision model.

What this version does

get_element_at(x, y) returns the uid of the element at the viewport-relative CSS-pixel coordinate, and refreshes the page snapshot. The returned uid is consumable by the existing uid-based tools (click, hover, fill, drag, upload_file, …).

Gated by the existing --experimentalVision flag (same as click_at).

Schema

{
  x: number,  // viewport-relative CSS px
  y: number,
}

Response

• Element uid: <uid>
• Refreshed snapshot (response.includeSnapshot()).

Behavior

• Hit-test uses elementFromPoint with open-shadow piercing and same-origin iframe descent.
• Element already in the a11y snapshot: returned directly.
• Element outside the a11y tree (e.g. a plain <div>): injected via the existing extraHandles mechanism (the same path third-party-developer tools use), so the returned uid resolves through McpPage.getElementByUid.

Limitations

• Cannot reach closed shadow roots (no JS or CDP API exposes them).
• Cannot reach cross-origin / OOPIF iframes — error message points at list_pages + select_page for switching into the inner target.

Background

The first revision of this PR returned a self-contained markdown descriptor with CSS modes, file output, and a CDP-backed matched-rules path. Per @OrKoN's review, that was out of scope for a get_element_at primitive — the tool should integrate with the snapshot/uid system the rest of the tools are built around. This rewrite drops mode/css/pierceShadow/filePath, removes the descriptor formatter, computed-style projection, matched-rules summary, 50KB outerHTML cap, and the CDP matched-CSS path (~1100 lines).

The previous descriptor-style implementation is preserved on backup/get-element-at-full-impl for reference; if a separate get_element_styles(uid) style tool is ever wanted, the building blocks are there.

Test plan

Real Chrome under withMcpContext:

• returns a uid for an element at coordinates + refreshes the snapshot (round-trip via getElementByUid confirms id matches)
• throws with self-healing message when coords are outside the viewport
• resolves a plain non-accessible <div> via the extraHandles fallback (uid round-trip)
• pierces open shadow roots (resolved uid → inner #inner button)
• descends into a same-origin iframe (resolved uid → inner #inner element)
• tool registration under --experimental-vision (tests/index.test.ts)
• disabled-flag error message (tests/e2e/chrome-devtools-commands.test.ts)
• Gemini eval scenario at scripts/eval_scenarios/get_element_at_test.ts

npm run typecheck clean. npm run test -- tests/tools/input.test.ts passes. npm run gen clean (auto-regenerated docs/tool-reference.md, src/bin/chrome-devtools-cli-options.ts, src/telemetry/tool_call_metrics.json).

CLA: still pending sign-off.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[OrKoN]

Thanks for the PR, this seems to do more than get_element_at (querying styles, storing data to files). get_element_at should just return an uid from the snapshot that matches the element at coordinates to integrate with the rest of tools. It should probably use browser's hit testing API and it would be more maintainable if corresponding CDP support is added first. Perhaps you could file a feature request first with details about the use case so that we could prioritize it based on the user feedback? I am also not sure if #268 has anything to do with the experimental vision mode.

[medyas]

Thanks for the steer @OrKoN — you were right that the original shape was doing too much. Pushed a rewrite (dadcac9) that scopes this down to a uid-returning primitive matching the rest of the tools:

• Returns: Element uid: <uid> + refreshed snapshot. The uid flows directly into click / hover / fill / etc.
• Schema: {x, y} — dropped mode, css, pierceShadow, filePath.
• Dropped: descriptor formatter, computed-style projection, matched-rules summary, 50KB outerHTML cap + temp-file path, full-descriptor JSON save, CDP CSS.getMatchedStylesForNode path. About 1100 lines.
• Hit-test: elementFromPoint with open-shadow piercing and same-origin iframe descent. Elements outside the a11y tree (plain divs) are injected via the existing extraHandles mechanism — the same path the third-party-developer tools use — so the uid is resolvable by McpPage.getElementByUid.
• Limits: closed shadow roots and cross-origin / OOPIF iframes return a self-healing error pointing at list_pages + select_page.

On the "browser hit-testing API + CDP support first" point — I went with elementFromPoint (browser's native primitive, no internal CDP plumbing) since DOM.getNodeForLocation → ElementHandle requires reaching into puppeteer internals and the practical limits (closed shadow / OOPIF) are the same either way. Happy to switch to CDP if you'd prefer.

On #268 — agree the linkage is loose. The issue thread covers a few mechanisms (coord→DOM, overlay UIDs / Set-of-Marks, DevTools selection bridge). This PR covers only the first; it doesn't close out the issue.