Aller au contenu

Bibliothèques · Référence

Web Fetch Toolkit

An Effect AI toolkit for safely fetching known public web pages and extracting bounded Markdown with Xberg.

Dernière mise à jour le 9 octobre 2026

Overview

webfetch-toolkit reads a known public HTTPS URL with Effect HttpClient and uses the Xberg-backed FileExtractionService to return bounded Markdown. It does not search for URLs or use a paid search provider.

WebFetchToolkit defines the fixed tool contract. WebFetchToolkitLayer(options?) supplies the implementation, validates limits in its Effect error channel, and acquires HTTP and extraction services during layer acquisition.

Install

shell
bunx shadcn@latest add @krak-stack/webfetch-toolkit

The item installs src/lib/webfetch-toolkit.ts and the Xberg-backed file extraction service. Package consumers import from @krak-stack/registry/webfetch-toolkit.

Compose With An Agent

ts
import { Effect, Layer } from "effect";
import { Toolkit } from "effect/ai";
import { FetchHttpClient } from "effect/http";
import { WebFetchToolkit, WebFetchToolkitLayer } from "@/lib/webfetch-toolkit";
import { FileExtractionService } from "@/services/file-extraction";

const toolkitDefinition = Toolkit.merge(application.toolkit, WebFetchToolkit);
const webFetchLayer = WebFetchToolkitLayer({
  maxResponseBytes: 5 * 1024 * 1024,
  maxContentCharacters: 50_000,
}).pipe(
  Layer.provide(FileExtractionService.layer),
  Layer.provide(FetchHttpClient.layer),
);
const toolsLayer = Layer.mergeAll(application.layer, webFetchLayer);
const toolkit = yield* toolkitDefinition.pipe(Effect.provide(toolsLayer));

const events = yield* agent.stream({
  action: request.action,
  history: request.history,
  systemPrompt: application.systemPrompt,
  toolkit,
});

application, agent, and request belong to the consuming application. Definitions remain available synchronously for inspection and composition, independently of handler dependencies.

Options

  • maxResponseBytes defaults to 2 MB and may be configured up to the Xberg-aligned 20 MB ceiling.
  • maxContentCharacters defaults to 30,000 and may be configured up to 200,000 characters.

Invalid options fail during handler layer acquisition rather than throwing synchronously. The tool uses a fixed 20-second request timeout.

Tool Contract

The model calls webFetch with one url. Successful results contain the original URL, normalized content type, extracted Markdown, and a truncated flag. Failures use invalid-response, too-large, unavailable, or unsupported-content.

Security

The request schema requires HTTPS, rejects URL credentials and custom ports, and blocks single-label/internal hostnames and literal IP addresses. Responses have time, byte, content-type, and model-context limits. Fetched content is untrusted data and must never authorize privileged actions.

Hostname validation cannot prevent a public DNS name from resolving to a private address or changing between resolution and connection. Production deployments should enforce outbound network policy through a proxy, firewall, or runtime that blocks loopback, private, link-local, and cloud metadata ranges, and should validate redirect destinations. Apply authentication, per-user quotas, and concurrency limits at the agent endpoint.

Dependencies
Packages and shadcn components required by this registry item.

Dependencies

Registry Dependencies