observe()

Overview

The observe() method identifies actionable elements on a page and returns Action objects that can be previewed, cached, or executed later with act(). This is useful for:

Previewing actions before execution
Caching actions to replay later
Hiding sensitive data from LLM inference
Building action libraries for testing

Method Signature

observe(instruction?: string, options?: ObserveOptions): Promise<Action[]>

Parameters

instruction

string

Natural language description of elements to find (e.g., “find the submit button”). If not provided, returns all actionable elements on the page.

options

ObserveOptions

Optional configuration for observation.

Show properties

model

ModelConfiguration

Override the default model for this specific observation.

timeout

number

Maximum time in milliseconds to wait for observation. Throws ObserveTimeoutError if exceeded.

selector

string

Focus observation on a specific part of the page. Accepts CSS selectors or XPath (prefix with xpath=).

page

Page

Specific page to observe (useful for multi-page scenarios).

Return Value

Returns a Promise<Action[]> - an array of Action objects:

interface Action {
  selector: string;      // XPath selector for the element
  description: string;   // Human-readable description
  method?: string;       // Playwright method to use (click, fill, etc.)
  arguments?: string[];  // Arguments for the method
}

Usage Examples

Basic Observation

import { Stagehand } from "@stagehand/api";

const stagehand = new Stagehand({
  env: "BROWSERBASE",
  apiKey: process.env.BROWSERBASE_API_KEY,
});

await stagehand.init();
const page = stagehand.context.pages()[0];

await page.goto("https://www.apartments.com/san-francisco-ca/");

// Observe a specific element
const [action] = await stagehand.observe("find the 'all filters' button");

console.log("Found:", action.description);
console.log("Selector:", action.selector);
console.log("Method:", action.method);

// Execute the observed action
await stagehand.act(action);

Preview Before Execution

// Find the action
const [loginAction] = await stagehand.observe(
  "find the login button"
);

// Review what will happen
console.log("Will execute:", loginAction.description);
console.log("Using method:", loginAction.method);

// Confirm with user or validate
const userConfirmed = await promptUser(
  `Execute ${loginAction.description}?`
);

if (userConfirmed) {
  await stagehand.act(loginAction);
}

Caching Actions

import fs from "fs/promises";

// Observe and cache actions
const actions = [
  await stagehand.observe("find the username field"),
  await stagehand.observe("find the password field"),
  await stagehand.observe("find the login button"),
];

// Save to file
await fs.writeFile(
  "login-actions.json",
  JSON.stringify(actions, null, 2)
);

// Later, load and execute without LLM inference
const cached = JSON.parse(
  await fs.readFile("login-actions.json", "utf-8")
);

for (const action of cached.flat()) {
  await stagehand.act(action);
}

Find All Actionable Elements

// Without instruction - returns all actionable elements
const allActions = await stagehand.observe();

console.log(`Found ${allActions.length} actionable elements:`);
allActions.forEach((action) => {
  console.log(`- ${action.description} (${action.method})`);
});

Focused Observation

// Observe only within a specific section
const sidebarActions = await stagehand.observe(
  "find navigation links",
  {
    selector: "aside.sidebar",
  }
);

// Or use XPath
const formActions = await stagehand.observe(
  "find all form fields",
  {
    selector: "xpath=//form[@id='contact']",
  }
);

Multi-Step Form Workflow

await page.goto("https://www.apartments.com/san-francisco-ca/");

// Observe each step
let observation: Action;

[observation] = await stagehand.observe(
  "find the 'all filters' button"
);
await stagehand.act(observation);
await new Promise((resolve) => setTimeout(resolve, 2000));

[observation] = await stagehand.observe(
  "find the '1+' button in the 'beds' section"
);
await stagehand.act(observation);
await new Promise((resolve) => setTimeout(resolve, 2000));

[observation] = await stagehand.observe(
  "find the 'apartments' button in the 'home type' section"
);
await stagehand.act(observation);
await new Promise((resolve) => setTimeout(resolve, 2000));

[observation] = await stagehand.observe(
  "find the pet policy dropdown"
);
await stagehand.act(observation);

Multi-Page Observation

const page1 = stagehand.context.pages()[0];
const page2 = await stagehand.context.newPage();

await page2.goto("https://example.com/form");

// Observe on specific page
const actions = await stagehand.observe(
  "find all form fields",
  { page: page2 }
);

Building Action Libraries

// Create reusable action library
class ActionLibrary {
  private actions: Map<string, Action> = new Map();

  async register(
    name: string,
    instruction: string,
    stagehand: Stagehand
  ) {
    const [action] = await stagehand.observe(instruction);
    this.actions.set(name, action);
  }

  get(name: string): Action | undefined {
    return this.actions.get(name);
  }

  async execute(name: string, stagehand: Stagehand) {
    const action = this.get(name);
    if (!action) throw new Error(`Action ${name} not found`);
    await stagehand.act(action);
  }
}

// Usage
const lib = new ActionLibrary();
await lib.register("login", "find login button", stagehand);
await lib.register("submit", "find submit button", stagehand);

// Execute by name
await lib.execute("login", stagehand);

With Timeout

try {
  const actions = await stagehand.observe(
    "find the button",
    { timeout: 15000 } // 15 seconds
  );
} catch (error) {
  if (error instanceof ObserveTimeoutError) {
    console.error("Observation timed out");
  }
}

Filtering Actions

const allActions = await stagehand.observe();

// Filter by method
const clickableElements = allActions.filter(
  (action) => action.method === "click"
);

const inputFields = allActions.filter(
  (action) => action.method === "fill"
);

console.log(`Found ${clickableElements.length} clickable elements`);
console.log(`Found ${inputFields.length} input fields`);

How It Works

Snapshot: Captures an accessibility tree of the page
LLM Processing: AI analyzes the tree to find actionable elements
Mapping: Converts element IDs to XPath selectors
Return: Returns Action objects with method and arguments

Action Methods

Observed actions can include these Playwright methods:

click - Click on element
fill - Type into input field
selectOption - Select from dropdown
check / uncheck - Toggle checkbox
hover - Hover over element
dragAndDrop - Drag to target (target in arguments)
press - Press keyboard key
not-supported - Element in shadow DOM or not actionable

Use Cases

1. Action Preview

Show users what will happen before executing:

const [action] = await stagehand.observe("find submit button");
console.log(`Ready to: ${action.description}`);
// User confirms, then execute
await stagehand.act(action);

2. Action Caching

Reduce LLM calls by caching actions:

// Cache during development
const actions = await stagehand.observe("find all buttons");
saveToCache(actions);

// Use in production without LLM
const cached = loadFromCache();
await stagehand.act(cached[0]);

3. Sensitive Data Handling

Keep credentials out of LLM context:

// Observe without sensitive data
const [usernameField] = await stagehand.observe(
  "find username field"
);
const [passwordField] = await stagehand.observe(
  "find password field"
);

// Execute with variables (not sent to LLM)
await stagehand.act(usernameField, {
  variables: { value: process.env.USERNAME }
});
await stagehand.act(passwordField, {
  variables: { value: process.env.PASSWORD }
});

4. Testing & Validation

Build test suites with observed actions:

const actions = await stagehand.observe();

// Validate expected elements exist
expect(actions.some(
  (a) => a.description.includes("submit")
)).toBe(true);

Best Practices

Be specific with instructions - Get targeted results

// Good
await stagehand.observe("find the blue submit button in the footer");

// Less specific
await stagehand.observe("find button");

Review before executing - Check action descriptions

const [action] = await stagehand.observe(instruction);
console.log("Will perform:", action.description);
await stagehand.act(action);

Cache for repeated workflows - Save on LLM costs

// Cache once
const actions = await observeAndCache();

// Reuse many times
for (let i = 0; i < 100; i++) {
  await stagehand.act(actions[0]);
}

Use focused observation - Improve accuracy on complex pages

await stagehand.observe(instruction, {
  selector: ".main-form"
});

Error Handling

try {
  const actions = await stagehand.observe("find button");
  
  if (actions.length === 0) {
    console.log("No actions found");
  } else if (actions[0].method === "not-supported") {
    console.log("Element not actionable");
  } else {
    await stagehand.act(actions[0]);
  }
} catch (error) {
  if (error instanceof ObserveTimeoutError) {
    console.error("Observation timed out");
  } else {
    console.error("Observation failed:", error);
  }
}

act() - Execute actions (including observed actions)
extract() - Extract data from pages
agent() - Autonomous multi-step automation

Getting Started

Core Concepts

Core Methods

Configuration

Integrations

Best Practices

Advanced Features

Overview

Method Signature

Parameters

Return Value

Usage Examples

Basic Observation

Preview Before Execution

Caching Actions

Find All Actionable Elements

Focused Observation

Multi-Step Form Workflow

Multi-Page Observation

Building Action Libraries

With Timeout

Filtering Actions

How It Works

Action Methods

Use Cases

1. Action Preview

2. Action Caching

3. Sensitive Data Handling

4. Testing & Validation

Best Practices

Error Handling

Getting Started

Core Concepts

Core Methods

Configuration

Integrations

Best Practices

Advanced Features

Documentation Index

​Overview

​Method Signature

​Parameters

​Return Value

​Usage Examples

​Basic Observation

​Preview Before Execution

​Caching Actions

​Find All Actionable Elements

​Focused Observation

​Multi-Step Form Workflow

​Multi-Page Observation

​Building Action Libraries

​With Timeout

​Filtering Actions

​How It Works

​Action Methods

​Use Cases

​1. Action Preview

​2. Action Caching

​3. Sensitive Data Handling

​4. Testing & Validation

​Best Practices

​Error Handling

​Related Methods

Overview

Method Signature

Parameters

Return Value

Usage Examples

Basic Observation

Preview Before Execution

Caching Actions

Find All Actionable Elements

Focused Observation

Multi-Step Form Workflow

Multi-Page Observation

Building Action Libraries

With Timeout

Filtering Actions

How It Works

Action Methods

Use Cases

1. Action Preview

2. Action Caching

3. Sensitive Data Handling

4. Testing & Validation

Best Practices

Error Handling

Related Methods