SkillHub / playwright-interactive

playwright-interactive

Persistent browser and Electron interaction through `js_repl` for fast iterative UI debugging.

Nameplaywright-interactive
Version1.0.0
Authorseekthought
Tagstesting playwright debugging interactive browser
CompatibilityRequires js_repl feature enabled in Codex config.

安装

skillctl install -r skillhub playwright-interactive

Playwright Interactive Skill

Use a persistent js_repl Playwright session to debug local web or Electron apps, keep the same handles alive across iterations, and run functional plus visual QA without restarting the whole toolchain unless the process ownership changed.

Preconditions

[features]
js_repl = true

One-time setup

test -f package.json || npm init -y
npm install playwright
# Web-only, for headed Chromium or mobile emulation:
# npx playwright install chromium
# Electron-only, and only if the target workspace is the app itself:
# npm install --save-dev electron
node -e "import('playwright').then(() => console.log('playwright import ok')).catch((error) => { console.error(error); process.exit(1); })"

If you switch to a different workspace later, repeat setup there.

Core Workflow

  1. Write a brief QA inventory before testing:
    • Build the inventory from three sources: the user's requested requirements, the user-visible features or behaviors you actually implemented, and the claims you expect to make in the final response.
    • Anything that appears in any of those three sources must map to at least one QA check before signoff.
    • List the user-visible claims you intend to sign off on.
    • List every meaningful user-facing control, mode switch, or implemented interactive behavior.
    • List the state changes or view changes each control or implemented behavior can cause.
    • Use this as the shared coverage list for both functional QA and visual QA.
    • For each claim or control-state pair, note the intended functional check, the specific state where the visual check must happen, and the evidence you expect to capture.
    • If a requirement is visually central but subjective, convert it into an observable QA check instead of leaving it implicit.
    • Add at least 2 exploratory or off-happy-path scenarios that could expose fragile behavior.
  2. Run the bootstrap cell once.
  3. Start or confirm any required dev server in a persistent TTY session.
  4. Launch the correct runtime and keep reusing the same Playwright handles.
  5. After each code change, reload for renderer-only changes or relaunch for main-process/startup changes.
  6. Run functional QA with normal user input.
  7. Run a separate visual QA pass.
  8. Verify viewport fit and capture the screenshots needed to support your claims.
  9. Clean up the Playwright session only when the task is actually finished.

Bootstrap (Run Once)

var chromium;
var electronLauncher;
var browser;
var context;
var page;
var mobileContext;
var mobilePage;
var electronApp;
var appWindow;

try {
  ({ chromium, _electron: electronLauncher } = await import("playwright"));
  console.log("Playwright loaded");
} catch (error) {
  throw new Error(
    `Could not load playwright from the current js_repl cwd. Run the setup commands from this workspace first. Original error: ${error}`
  );
}

Binding rules:

Shared web helpers:

var resetWebHandles = function () {
  context = undefined;
  page = undefined;
  mobileContext = undefined;
  mobilePage = undefined;
};

var ensureWebBrowser = async function () {
  if (browser && !browser.isConnected()) {
    browser = undefined;
    resetWebHandles();
  }

  browser ??= await chromium.launch({ headless: false });
  return browser;
};

var reloadWebContexts = async function () {
  for (const currentContext of [context, mobileContext]) {
    if (!currentContext) continue;
    for (const p of currentContext.pages()) {
      await p.reload({ waitUntil: "domcontentloaded" });
    }
  }
  console.log("Reloaded existing web tabs");
};

Choose Session Mode

For web apps, use an explicit viewport by default and treat native-window mode as a separate validation pass.

Start or Reuse Web Session

Desktop and mobile web sessions share the same browser, helpers, and QA flow. The main difference is which context and page pair you create.

Desktop Web Context

Set TARGET_URL to the app you are debugging. For local servers, prefer 127.0.0.1 over localhost.

var TARGET_URL = "http://127.0.0.1:3000";

if (page?.isClosed()) page = undefined;

await ensureWebBrowser();
context ??= await browser.newContext({
  viewport: { width: 1600, height: 900 },
});
page ??= await context.newPage();

await page.goto(TARGET_URL, { waitUntil: "domcontentloaded" });
console.log("Loaded:", await page.title());

If context or page is stale, set context = page = undefined and rerun the cell.

Mobile Web Context

Reuse TARGET_URL when it already exists; otherwise set a mobile target directly.

var MOBILE_TARGET_URL = typeof TARGET_URL === "string"
  ? TARGET_URL
  : "http://127.0.0.1:3000";

if (mobilePage?.isClosed()) mobilePage = undefined;

await ensureWebBrowser();
mobileContext ??= await browser.newContext({
  viewport: { width: 390, height: 844 },
  isMobile: true,
  hasTouch: true,
});
mobilePage ??= await mobileContext.newPage();

await mobilePage.goto(MOBILE_TARGET_URL, { waitUntil: "domcontentloaded" });
console.log("Loaded mobile:", await mobilePage.title());

If mobileContext or mobilePage is stale, set mobileContext = mobilePage = undefined and rerun the cell.

Native-Window Web Pass

var TARGET_URL = "http://127.0.0.1:3000";

await ensureWebBrowser();

await page?.close().catch(() => {});
await context?.close().catch(() => {});
page = undefined;
context = undefined;

browser ??= await chromium.launch({ headless: false });
context = await browser.newContext({ viewport: null });
page = await context.newPage();

await page.goto(TARGET_URL, { waitUntil: "domcontentloaded" });
console.log("Loaded native window:", await page.title());

Start or Reuse Electron Session

Set ELECTRON_ENTRY to . when the current workspace is the Electron app and package.json points main to the right entry file. If you need to target a specific main-process file directly, use a path such as ./main.js instead.

var ELECTRON_ENTRY = ".";

if (appWindow?.isClosed()) appWindow = undefined;

if (!appWindow && electronApp) {
  await electronApp.close().catch(() => {});
  electronApp = undefined;
}

electronApp ??= await electronLauncher.launch({
  args: [ELECTRON_ENTRY],
});

appWindow ??= await electronApp.firstWindow();

console.log("Loaded Electron window:", await appWindow.title());

If js_repl is not already running from the Electron app workspace, pass cwd explicitly when launching.

If the app process looks stale, set electronApp = appWindow = undefined and rerun the cell.

If you already have an Electron session but need a fresh process after a main-process, preload, or startup change, use the restart cell in the next section instead of rerunning this one.

Reuse Sessions During Iteration

Keep the same session alive whenever you can.

Web renderer reload:

await reloadWebContexts();

Electron renderer-only reload:

await appWindow.reload({ waitUntil: "domcontentloaded" });
console.log("Reloaded Electron window");

Electron restart after main-process, preload, or startup changes:

await electronApp.close().catch(() => {});
electronApp = undefined;
appWindow = undefined;

electronApp = await electronLauncher.launch({
  args: [ELECTRON_ENTRY],
});

appWindow = await electronApp.firstWindow();
console.log("Relaunched Electron window:", await appWindow.title());

If your launch requires an explicit cwd, include the same cwd here.

Default posture:

Checklists

Session Loop

Reload Decision

Functional QA

Visual QA

Signoff

Screenshot Examples

If you plan to emit a screenshot through codex.emitImage(...), use the CSS-normalized paths in the next section by default. Those are the canonical examples for screenshots that will be interpreted by the model or used for coordinate-based follow-up actions. Keep raw captures as an exception for fidelity-sensitive debugging only; the raw exception examples appear after the normalization guidance.

Model-bound screenshots (default)

If you will emit a screenshot with codex.emitImage(...) for model interpretation, normalize it to CSS pixels for the exact region you captured before emitting. This keeps returned coordinates aligned with Playwright CSS pixels if the reply is later used for clicking, and it also reduces image payload size and model token cost.

Do not emit raw native-window screenshots by default. Skip normalization only when you explicitly need device-pixel fidelity, such as Retina or DPI artifact debugging, pixel-accurate rendering inspection, or another fidelity-sensitive case where raw pixels matter more than payload size. For local-only inspection that will not be emitted to the model, raw capture is fine.

Do not assume page.screenshot({ scale: "css" }) is enough in native-window mode (viewport: null). In Chromium on macOS Retina displays, headed native-window screenshots can still come back at device-pixel size even when scale: "css" is requested. The same caveat applies to Electron windows launched through Playwright because Electron runs with noDefaultViewport, and appWindow.screenshot({ scale: "css" }) may still return device-pixel output.

Use separate normalization paths for web pages and Electron windows:

Shared helpers and conventions:

var emitJpeg = async function (bytes) {
  await codex.emitImage({
    bytes,
    mimeType: "image/jpeg",
    detail: "original",
  });
};

var emitWebJpeg = async function (surface, options = {}) {
  await emitJpeg(await surface.screenshot({
    type: "jpeg",
    quality: 85,
    scale: "css",
    ...options,
  }));
};

var clickCssPoint = async function ({ surface, x, y, clip }) {
  await surface.mouse.click(
    clip ? clip.x + x : x,
    clip ? clip.y + y : y
  );
};

var tapCssPoint = async function ({ page, x, y, clip }) {
  await page.touchscreen.tap(
    clip ? clip.x + x : x,
    clip ? clip.y + y : y
  );
};

Web CSS normalization

Preferred web path for explicit-viewport contexts, and often for web in general:

await emitWebJpeg(page);

Mobile web uses the same path; substitute mobilePage for page:

await emitWebJpeg(mobilePage);

If the model returns { x, y }, click it directly:

await clickCssPoint({ surface: page, x, y });

Mobile web click path:

await tapCssPoint({ page: mobilePage, x, y });

For web clip screenshots or element screenshots in this normal path, scale: "css" usually works directly. Add the region origin back when clicking.

Web native-window fallback when scale: "css" still comes back at device-pixel size:

var emitWebScreenshotCssScaled = async function ({ page, clip, quality = 0.85 } = {}) {
  var NodeBuffer = (await import("node:buffer")).Buffer;
  const target = clip
    ? { width: clip.width, height: clip.height }
    : await page.evaluate(() => ({
        width: window.innerWidth,
        height: window.innerHeight,
      }));

  const screenshotBuffer = await page.screenshot({
    type: "png",
    ...(clip ? { clip } : {}),
  });

  const bytes = await page.evaluate(
    async ({ imageBase64, targetWidth, targetHeight, quality }) => {
      const image = new Image();
      image.src = `data:image/png;base64,${imageBase64}`;
      await image.decode();

      const canvas = document.createElement("canvas");
      canvas.width = targetWidth;
      canvas.height = targetHeight;

      const ctx = canvas.getContext("2d");
      ctx.imageSmoothingEnabled = true;
      ctx.drawImage(image, 0, 0, targetWidth, targetHeight);

      const blob = await new Promise((resolve) =>
        canvas.toBlob(resolve, "image/jpeg", quality)
      );

      return new Uint8Array(await blob.arrayBuffer());
    },
    {
      imageBase64: NodeBuffer.from(screenshotBuffer).toString("base64"),
      targetWidth: target.width,
      targetHeight: target.height,
      quality,
    }
  );

  await emitJpeg(bytes);
};

For a full viewport fallback capture, treat returned { x, y } as direct CSS coordinates:

await emitWebScreenshotCssScaled({ page });
await clickCssPoint({ surface: page, x, y });

For a clipped fallback capture, add the clip origin back:

await emitWebScreenshotCssScaled({ page, clip });
await clickCssPoint({ surface: page, clip, x, y });

Electron CSS normalization

For Electron, normalize in the main process instead of opening a scratch Playwright page. The helper below returns CSS-scaled bytes for the full content area or for a clipped CSS-pixel region. Treat clip as content-area CSS pixels, for example values taken from getBoundingClientRect() in the renderer.

var emitElectronScreenshotCssScaled = async function ({ electronApp, clip, quality = 85 } = {}) {
  const bytes = await electronApp.evaluate(async ({ BrowserWindow }, { clip, quality }) => {
    const win = BrowserWindow.getAllWindows()[0];
    const image = clip ? await win.capturePage(clip) : await win.capturePage();

    const target = clip
      ? { width: clip.width, height: clip.height }
      : (() => {
          const [width, height] = win.getContentSize();
          return { width, height };
        })();

    const resized = image.resize({
      width: target.width,
      height: target.height,
      quality: "best",
    });

    return resized.toJPEG(quality);
  }, { clip, quality });

  await emitJpeg(bytes);
};

Full Electron window:

await emitElectronScreenshotCssScaled({ electronApp });
await clickCssPoint({ surface: appWindow, x, y });

Clipped Electron region using CSS pixels from the renderer:

var clip = await appWindow.evaluate(() => {
  const rect = document.getElementById("board").getBoundingClientRect();
  return {
    x: Math.round(rect.x),
    y: Math.round(rect.y),
    width: Math.round(rect.width),
    height: Math.round(rect.height),
  };
});

await emitElectronScreenshotCssScaled({ electronApp, clip });
await clickCssPoint({ surface: appWindow, clip, x, y });

Raw Screenshot Exception Examples

Use these only when raw pixels matter more than CSS-coordinate alignment, such as Retina or DPI artifact debugging, pixel-accurate rendering inspection, or other fidelity-sensitive review.

Web desktop raw emit:

await codex.emitImage({
  bytes: await page.screenshot({ type: "jpeg", quality: 85 }),
  mimeType: "image/jpeg",
  detail: "original",
});

Electron raw emit:

await codex.emitImage({
  bytes: await appWindow.screenshot({ type: "jpeg", quality: 85 }),
  mimeType: "image/jpeg",
  detail: "original",
});

Mobile raw emit after the mobile web context is already running:

await codex.emitImage({
  bytes: await mobilePage.screenshot({ type: "jpeg", quality: 85 }),
  mimeType: "image/jpeg",
  detail: "original",
});

Viewport Fit Checks (Required)

Do not assume a screenshot is acceptable just because the main widget is visible. Before signoff, explicitly verify that the intended initial view matches the product requirement, using both screenshot review and numeric checks.

Web or renderer check:

console.log(await page.evaluate(() => ({
  innerWidth: window.innerWidth,
  innerHeight: window.innerHeight,
  clientWidth: document.documentElement.clientWidth,
  clientHeight: document.documentElement.clientHeight,
  scrollWidth: document.documentElement.scrollWidth,
  scrollHeight: document.documentElement.scrollHeight,
  canScrollX: document.documentElement.scrollWidth > document.documentElement.clientWidth,
  canScrollY: document.documentElement.scrollHeight > document.documentElement.clientHeight,
})));

Electron check:

console.log(await appWindow.evaluate(() => ({
  innerWidth: window.innerWidth,
  innerHeight: window.innerHeight,
  clientWidth: document.documentElement.clientWidth,
  clientHeight: document.documentElement.clientHeight,
  scrollWidth: document.documentElement.scrollWidth,
  scrollHeight: document.documentElement.scrollHeight,
  canScrollX: document.documentElement.scrollWidth > document.documentElement.clientWidth,
  canScrollY: document.documentElement.scrollHeight > document.documentElement.clientHeight,
})));

Augment the numeric check with getBoundingClientRect() checks for the required visible regions in your specific UI when clipping is a realistic failure mode; document-level metrics alone are not sufficient for fixed shells.

Dev Server

For local web debugging, keep the app running in a persistent TTY session. Do not rely on one-shot background commands from a short-lived shell.

Use the project's normal start command, for example:

npm start

Before page.goto(...), verify the chosen port is listening and the app responds.

For Electron debugging, launch the app from js_repl through _electron.launch(...) so the same session owns the process. If the Electron renderer depends on a separate dev server (for example Vite or Next), keep that server running in a persistent TTY session and then relaunch or reload the Electron app from js_repl.

Cleanup

Only run cleanup when the task is actually finished:

if (electronApp) {
  await electronApp.close().catch(() => {});
}

if (mobileContext) {
  await mobileContext.close().catch(() => {});
}

if (context) {
  await context.close().catch(() => {});
}

if (browser) {
  await browser.close().catch(() => {});
}

browser = undefined;
context = undefined;
page = undefined;
mobileContext = undefined;
mobilePage = undefined;
electronApp = undefined;
appWindow = undefined;

console.log("Playwright session closed");

If you plan to exit Codex immediately after debugging, run the cleanup cell first and wait for the "Playwright session closed" log before quitting.

Common Failure Modes