`, drop nav/footer), convert to markdown. 2. **Tier 2** (only if Tier 1 returns very little): use Playwright service to render and extract. Save the result to disk and write a row in the sqlite cache. Subsequent calls will hit the cache. ### Step 7: Tool returns to the agent The result is wrapped in an "untrusted content" envelope so the existing prompt-injection scrubbing applies: ``` [BEGIN UNTRUSTED TRADECRAFT RESULT] resource: hacktricks url: https://book.hacktricks.wiki/.../kerberoast.html section_title: Kerberoast fetched_at: 2026-04-26T14:02:11Z (cache miss, tier 1) --- Code blocks: - bash: impacket-GetUserSPNs domain.local/user:password ... - powershell: Invoke-Kerberoast | Out-File hashes.txt [END UNTRUSTED TRADECRAFT RESULT] ``` ### Step 8: Agent uses the content The agent now has the actual Kerberoasting page content in its context. It reads the technique, picks the relevant commands, and proceeds with the attack chain. Maybe it generates a kali shell command using `impacket-GetUserSPNs`, then asks for the next step. If it later needs another technique ("now I have a Kerberos ticket, how do I crack it?"), it calls the tool again. Same flow. Cache may hit if the page was already fetched recently. ## Agent session flow (sequence diagram) ```mermaid sequenceDiagram participant User participant Agent as Claude (main agent) participant Tool as tradecraft_lookup participant Cache as sqlite + disk cache participant Web as Target Website participant LLM as Claude (small section pick) User->>Agent: "How do I escalate to Domain Admin?" Note over Agent: Reads tool catalog,
matches HackTricks summary Agent->>Tool: tradecraft_lookup(
resource_id="hacktricks",
query="...") Tool->>Tool: Look up resource by id Tool->>Tool: Lexical rank sitemap (top 30) Tool->>LLM: Pick best of 30 titles LLM-->>Tool: "Kerberoast" Tool->>Cache: Lookup URL alt Cache hit & fresh Cache-->>Tool: cached markdown else Cache miss or stale Tool->>Web: HTTP GET (Tier 1) Web-->>Tool: HTML Tool->>Tool: Parse, extract article, markdown Tool->>Cache: Save to disk + sqlite row end Tool-->>Agent: [UNTRUSTED RESULT envelope] Agent->>Agent: Read content, plan next move Agent-->>User: Suggests impacket-GetUserSPNs ... ``` --- # Special case: CVE PoC lookups CVE PoC repositories like `trickest/cve` and `0xMarcio/cve` are special. They have **hundreds of thousands of files**, organized deterministically by CVE-ID: ``` trickest/cve/ 2021/ CVE-2021-41773.md CVE-2021-44228.md ... 2022/ ... ``` There is no point in enumerating the tree (waste of time and storage) or running a section picker (the lookup key is exact). So the tool special-cases this type: ```mermaid flowchart TD A[Agent calls tool with
resource_id='trickest_cve'
cve_id='CVE-2021-41773'] --> B[Tool sees type=cve-poc-db] B --> C[Skip section picker entirely] C --> D[Build path: contents/2021/CVE-2021-41773.md] D --> E[GitHub Contents API direct hit] E --> F{200 OK?} F -->|yes| G[Return markdown] F -->|no| H[Fallback: list /contents/2021,
substring match CVE-ID] H --> I{found?} I -->|yes| G I -->|no| J[Return 'PoC not found'] ``` The agent's prompt sees this special requirement explicitly: the tool description for `cve-poc-db` resources says **"You MUST pass cve_id="CVE-YYYY-NNNNN""**. Claude knows to provide the ID when calling. --- # The cache layer All fetched pages are cached on disk. The cache is **per-resource and per-URL**, with TTLs that vary by resource type: | Resource type | Default TTL | Why | |---|---|---| | `mkdocs-wiki` | 7 days | Wikis update slowly | | `github-repo` | 7 days | Repos update slowly | | `cve-poc-db` | 30 days | PoCs are immutable once published | | `sphinx-docs` | 14 days | Tool docs change rarely | | `gitbook` | 7 days | Same as wikis | | `agentic-crawl` | 1 day | Generic blogs may update often | Per-resource override available via `cacheTtlSec` field. ```mermaid flowchart LR Q[Tool query] --> CC{Cache lookup} CC -->|hit & not expired| H[Return cached file
~ms] CC -->|miss or expired| F[Fetch live] F --> S[Save to disk + sqlite] S --> R[Return content
~seconds] style H fill:#e8e8e8,stroke:#888,stroke-width:2px,color:#222 style R fill:#d8d8d8,stroke:#777,stroke-width:1.5px,stroke-dasharray:3 3,color:#222 ``` **Where it lives**: `/app/tradecraft_cache//.md` on a Docker volume. SQLite at `/app/tradecraft_cache/index.sqlite` tracks `{url, fetched_at, ttl, resource_id, file_path, tier}`. **Force refresh**: pass `force_refresh=True` in the tool call, or click "Refresh" in the UI to invalidate the cache for that resource. --- # The agentic-crawl fallback When you add a URL that does not match any of the 5 deterministic types (a personal blog, a custom CMS, an internal wiki without published sitemap), the system falls back to **agentic-crawl**: a bounded LLM-driven loop that explores the site with Playwright and builds a sitemap. This runs **only at verify time**, never at query time. So it is slow once (90-180 seconds), then forever fast. ## How the loop works ```mermaid flowchart TD Start[Start at homepage] --> Fetch[Fetch page with Playwright] Fetch --> Extract[Extract title + same-host links
+ structural hints] Extract --> Filter[Filter out junk:
login, privacy, .png, etc.] Filter --> Add[Add to sitemap if
page has > 500 chars] Add --> Ask[Ask Claude:
which links to follow next?
or stop?] Ask --> Decision{Claude says...} Decision -->|follow N links| Enqueue[Add to frontier] Decision -->|stop| Done[Return sitemap] Enqueue --> Bounds{Bounds OK?
pages, time, calls, depth} Bounds -->|yes| Fetch Bounds -->|no| Done style Done fill:#e8e8e8,stroke:#888,stroke-width:2px,color:#222 style Bounds fill:#c8c8c8,stroke:#555,stroke-width:2px,color:#222 ``` ## The bounds (always enforced) | Bound | Default | Setting name | |---|---|---| | Max pages visited | 30 | `TRADECRAFT_CRAWL_MAX_PAGES` | | Max LLM calls | 20 | `TRADECRAFT_CRAWL_MAX_LLM_CALLS` | | Wall-clock budget | 180s | `TRADECRAFT_CRAWL_TIME_BUDGET_SEC` | | Max link depth from home | 3 | `TRADECRAFT_CRAWL_MAX_DEPTH` | If any single bound trips, the loop exits and returns whatever sitemap it built. The reason is shown in the UI badge: *"Crawled 27 pages, hit time budget"*. ## What the LLM sees per iteration For each page the loop visits, Claude gets a short prompt with: - The current page URL and title - The list of unvisited same-host links (top 20, with anchor text) - A small sample of what is already in the sitemap - Remaining budget - Page hints (search box present? pagination? tag cloud?) Claude returns JSON: either `{"action": "stop"}` or `{"action": "follow", "indices": [1, 4, 7]}`. ## Cost envelope Per verify of an agentic-crawl resource: - **Time**: 90 to 180 seconds - **LLM calls**: 15 to 25 - **Cost**: about $0.30 to $0.60 (with Sonnet) Paid once when you add the resource. Query-time stays sub-second because the sitemap is then just JSON in the database. --- # Putting it all together ## Lifecycle of one tradecraft resource ```mermaid stateDiagram-v2 [*] --> Adding: User clicks Add Resource Adding --> Verifying: Webapp creates row,
calls /verify Verifying --> Active: Summary + type + sitemap saved Verifying --> ActiveDegraded: Verify failed,
resource saved with lastError Active --> ToolCallable: Agent loads project ActiveDegraded --> ToolCallable: Same, but with warning ToolCallable --> CacheMiss: Agent calls tool, no cache ToolCallable --> CacheHit: Agent calls tool, cache hit CacheMiss --> ToolCallable: Page fetched + cached CacheHit --> ToolCallable: Page returned ~ms Active --> Refreshing: User clicks Refresh Refreshing --> Active: Sitemap rebuilt Active --> Disabled: User toggles off Disabled --> Active: User toggles on Disabled --> [*]: User deletes Active --> [*]: User deletes ``` ## What happens when a resource is disabled The tool's docstring is rebuilt every time a project loads its settings. Disabled resources are filtered out **before** the docstring is built, so the agent literally cannot see them as options. There is no risk of the agent trying to call a disabled resource. If the agent has a stale conversation and tries to call a slug that no longer exists (e.g. you deleted it), the tool returns an error envelope: `Resource '' not configured`. The agent recovers and tries something else. --- # Comparison with existing tools The tradecraft tool is intentionally separate from the existing `web_search` and the FAISS knowledge base. They serve different needs: | Tool | When to use | Source freshness | Latency | |---|---|---|---| | `web_search` | General research, open-web context, recent news | Real-time (Tavily) + pre-ingested KB | 1-3s | | FAISS KB (inside web_search) | Pre-ingested security corpora (gtfobins, lolbas, owasp, nvd, exploitdb, nuclei) | Stale unless re-ingested | <1s | | `tradecraft_lookup` | Specific exploitation technique, payload, or PoC from a trusted curated source | Always fresh (live fetch with cache) | <1s on cache hit, 1-5s on miss | The agent's prompt is updated to tell it: **prefer `tradecraft_lookup` after `query_graph` and `web_search` when you need a specific exploitation page or PoC, not general background.** --- # Summary of the three moving parts | Part | When it runs | What it does | |---|---|---| | **Verify** | Once when you click Save (and on Refresh) | Downloads homepage, detects type, builds sitemap, asks Claude for a summary. Saves all of it to the database row. Slow but durable. | | **Tool description build** | Every time the agent loads a project | Reads enabled resources, composes a text description listing each one with its saved summary. Hands it to Claude as the tool's documentation. Instant. | | **Tool call** | When Claude decides to call the tool during chat | Picks the right page from the sitemap, fetches it (or hits cache), returns content to Claude. Sub-second on cache hit. | The summary you save at add-time is **the entire intelligence behind which site Claude picks**. The sitemap you save at add-time is **what lets the tool find the right page without re-crawling**. Everything else is plumbing. --- # Glossary - **Tradecraft**: applied operator knowledge for offensive security. The "how to actually do it" between theory and tools. Originally a CIA term for spy operational skills. - **Resource**: one URL you added to the catalog (e.g. HackTricks). - **Resource type**: one of 6 categories (`mkdocs-wiki`, `github-repo`, `cve-poc-db`, `sphinx-docs`, `gitbook`, `agentic-crawl`) that determines how the system extracts a sitemap. - **Sitemap**: the per-resource map of pages, stored as JSON in the database. Built once at verify time. - **Catalog**: the dynamic tool description built from all enabled resources, shown to the agent at runtime. - **Verify**: the one-time process of fetching, typing, sitemap-building, and summarizing a newly-added resource. - **Section picker**: the at-query-time logic that narrows a sitemap to one page URL using lexical ranking + an optional small Claude call. - **Tier 1 / Tier 2**: two-step content fetch. Tier 1 is plain HTTP, Tier 2 is Playwright. Used both at verify time and query time.