# Flow Execution in PentAGI

This document describes the internal architecture and execution workflow of Flow in PentAGI, an autonomous penetration testing system that leverages AI agents to perform complex security testing workflows.

## 1. Core Concepts and Terminology

### Hierarchy
- **Flow** - Top-level workflow representing a complete penetration testing session (persistent)
- **Task** - User-defined objective within a Flow (multiple Tasks can exist in one Flow)
- **Subtask** - Auto-decomposed sequential step to complete a Task (generated and refined by system)
- **Action** - Individual operation performed by agents (commands, searches, analyses)

### Workers
- **FlowWorker** - Manages the complete lifecycle of a Flow, coordinates Tasks
- **TaskWorker** - Executes individual Tasks, manages Subtask generation and refinement
- **SubtaskWorker** - Handles execution of specific Subtasks via AI agents
- **AssistantWorker** - Manages interactive assistant mode within a Flow; optionally controls the Flow's automation via injected `FlowWorker` reference

### Providers
- **FlowProvider** - Core interface for Flow execution, agent coordination and orchestration
- **AssistantProvider** - Specialized provider for assistant mode interactions; receives `FlowWorker` via `SetFlowWorker()` after chain preparation to enable flow management tool callbacks
- **ProviderController** - Factory for creating and managing different LLM providers

### AI Agents
- **Primary Agent** - Main orchestrator that coordinates all other agents within a Subtask
- **Generator Agent** - Decomposes Tasks into ordered lists of Subtasks (max 15)
- **Refiner Agent** - Reviews and updates planned Subtask list after each Subtask completion (can add/remove/modify planned Subtasks)
- **Reporter Agent** - Creates comprehensive final reports for completed Tasks
- **Coder Agent** - Writes and maintains code for specific requirements
- **Pentester Agent** - Performs penetration testing and vulnerability assessment
- **Installer Agent** - Manages environment setup and tool installation
- **Memorist Agent** - Handles long-term memory storage and retrieval
- **Searcher Agent** - Conducts internet research and information gathering
- **Enricher Agent** - Enhances information from multiple sources
- **Adviser Agent** - Provides expert guidance and recommendations
- **Reflector Agent** - Corrects agents that return unstructured text instead of tool calls
- **Assistant Agent** - Provides interactive assistance, operates autonomously within Flow independently from Task/Subtask (UseAgents flag controls delegation); when FlowWorker is available, exposes `get_flow_status`, `stop_flow`, `submit_flow_input`, and `patch_flow_subtasks` tools for automation control

### Tools and Capabilities by Category
- **Environment Tools** - Terminal commands, file operations within Docker containers
  - `terminal` - Command execution (configurable via `TERMINAL_TOOL_TIMEOUT`, default: 1200s; hard limit: 3h/10800s; 0 or negative values are clamped to the hard limit)
  - `file` - Read/write operations with absolute path requirements
  - User-provided flow files are available under `/work/uploads` and `/work/resources`
  
- **Search Network Tools** - External information sources
  - `browser` - Web scraping with screenshot capture
  - `web_search` - Unified web search. The agent supplies a `query` and an intent `mode`
    (`links`, `answer`, `research`, `exploit`); the orchestrator selects the underlying
    engine, retries transient failures, and falls back across providers automatically.
    Engines orchestrated under the hood (when configured): Google, DuckDuckGo, Searxng,
    Tavily, Traversaal, Perplexity, Firecrawl, Sploitus, and the optional internal
    browser-analytics engine (`WEB_SEARCH_INTERNAL_ENABLED`, off by default). The
    per-mode fallback order is a single data table at the top of
    `backend/pkg/tools/web_search.go`; the engine primitives live in
    `backend/pkg/tools/searchers/`.
  
- **Vector Database Tools** - Semantic search in long-term memory
  - `search_in_memory` - General execution memory search
  - `search_guide` / `store_guide` - Installation guides (doc_type: guide)
  - `search_answer` / `store_answer` - Q&A pairs (doc_type: answer)
  - `search_code` / `store_code` - Code samples (doc_type: code)
  
- **Agent Tools** - Delegation to specialist agents
  - `search`, `maintenance`, `coder`, `pentester`, `advice`, `memorist`
  
- **Result Storage Tools** - Agent result delivery
  - `maintenance_result`, `code_result`, `hack_result`, `memorist_result`
  - `search_result`, `enricher_result`, `report_result`
  - `subtask_list` (Generator), `subtask_patch` (Refiner)
  - Future signed evidence receipt design is outlined in [evidence_chain.md](../../examples/proposals/evidence_chain.md)

- **Barrier Tools** - Control flow termination  
  - `done` - Complete subtask, `ask` - Request user input (configurable via ASK_USER env)

- **Flow Management Tools** - Assistant-only; available when FlowWorker is injected into AssistantProvider
  - `get_flow_status` - Query flow state, tasks, subtasks, planned/running subtask with optional agent messages (verbose mode: 50 messages, execution context)
  - `stop_flow` - Cancel the currently running task (15s timeout; reports actual post-stop state)
  - `submit_flow_input` - Deliver text to a waiting flow: answers an `ask` checkpoint or creates a new task
  - `patch_flow_subtasks` - Replace planned subtask list via delta operations (add/remove/modify/reorder); returns new IDs after recreation

### Execution Context
- **Message Chain** - Conversation history maintained for each agent interaction
- **Execution Context** - Comprehensive state including completed/planned Subtasks
- **Docker Environment** - Isolated container for secure tool execution
- **Vector Store** - Long-term semantic memory for knowledge retention
- **Task Materials** - Optional user uploads and resources exposed to prompts via `{{.UserFiles}}`

### Performance Results
- **PerformResultDone** - Subtask completed successfully via `done` tool
- **PerformResultWaiting** - Subtask paused for user input via `ask` tool
- **PerformResultError** - Subtask failed due to unrecoverable errors

## 2. Main Flow Execution Process

```mermaid
sequenceDiagram
    participant U as User
    participant FC as FlowController
    participant FW as FlowWorker
    participant FP as FlowProvider
    participant Docker as Docker Container
    participant TW as TaskWorker
    participant STC as SubtaskController
    participant SW as SubtaskWorker
    participant PA as Primary Agent
    participant GA as Generator Agent
    participant RA as Refiner Agent
    participant Reflector as Reflector Agent
    participant Rep as Reporter Agent
    participant DB as Database

    U->>FC: Submit penetration testing request
    FC->>FW: Create new Flow
    FW->>FP: Initialize FlowProvider
    FP->>FP: Call Image Chooser (select Docker image)
    FP->>FP: Call Language Chooser (detect user language)
    FP->>FP: Call Flow Descriptor (generate Flow title)
    FW->>Docker: Spawn container from selected image
    Docker-->>FW: Container ready
    
    FW->>TW: Create first Task
    TW->>FP: Get task title from user input
    FP-->>TW: Generated title
    TW->>DB: Store Task in database
    TW->>STC: Generate Subtasks
    STC->>GA: Invoke Generator Agent
    GA->>GA: Analyze task requirements
    GA-->>STC: Return Subtask list
    STC->>DB: Store Subtasks in database
    
    loop For each Subtask
        TW->>SW: Pop next Subtask
        SW->>FP: Prepare agent chain
        FP->>DB: Store message chain
        SW->>PA: Execute Primary Agent
        PA->>PA: Evaluate Subtask requirements
        
        alt Needs specialist agent
            PA->>Coder/Pentester/etc: Delegate specialized work
            Coder/Pentester/etc-->>PA: Return results
        end
        
        alt Agent returns unstructured text
            PA->>Reflector: Invoke Reflector Agent
            Reflector->>PA: Provide corrective guidance
            Note over Reflector: Acts as user, max 3 iterations
        end
        
        alt Needs user input
            PA->>PA: Call ask tool
            PA-->>U: Ask question
            U-->>PA: Provide answer
        end
        
        alt Subtask completed
            PA->>PA: Call done tool
            PA-->>SW: PerformResultDone
        end
        
        SW->>RA: Invoke Refiner Agent
        RA->>RA: Review completed/planned Subtasks
        RA->>DB: Update Subtask plans
        RA-->>TW: Updated planning
    end
    
    TW->>Rep: Generate final Task report
    Rep->>Rep: Analyze all Subtask results
    Rep-->>TW: Comprehensive report
    TW->>DB: Store Task result
    TW-->>U: Present final results
    
    opt New Task in same Flow
        U->>FW: Submit additional request
        FW->>TW: Create new Task (reuse experience)
        Note over TW: Process continues from Subtask generation
    end
```

## 3. AI Agent Interactions and Capabilities

```mermaid
graph TD
    subgraph "Primary Execution Flow"
        PA[Primary Agent<br/>Orchestrator] --> Coder[Coder Agent<br/>Development Specialist]
        PA --> Pentester[Pentester Agent<br/>Security Testing Specialist]
        PA --> Installer[Installer Agent<br/>Infrastructure Maintenance]
        PA --> Memorist[Memorist Agent<br/>Long-term Memory Specialist]
        PA --> Searcher[Searcher Agent<br/>Information Retrieval Specialist]
        PA --> Adviser[Adviser Agent<br/>Technical Solution Expert]
    end
    
    subgraph "Assistant Modes"
        AssistantUA[Assistant Agent<br/>UseAgents=true] --> Coder
        AssistantUA --> Pentester
        AssistantUA --> Installer
        AssistantUA --> Memorist
        AssistantUA --> Searcher
        AssistantUA --> Adviser
        
        AssistantDirect[Assistant Agent<br/>UseAgents=false] --> DirectTools[Direct Tools Only]
        
        AssistantNote[Operates independently<br/>from Task/Subtask hierarchy]
    end
    
    subgraph "Specialist Agent Tools"
        Coder --> Terminal[Terminal Tool]
        Coder --> File[File Tool]
        Coder --> CodeSearch[Search/Store Code]
        
        Pentester --> Terminal
        Pentester --> File
        Pentester --> Browser[Browser Tool]
        Pentester --> GuideSearch[Search/Store Guides]
        
        Installer --> Terminal
        Installer --> File
        Installer --> Browser
        Installer --> GuideSearch
        
        Memorist --> Terminal
        Memorist --> File
        Memorist --> VectorDB[Vector Database<br/>Memory Search]
    end
    
    subgraph "Search Tool Hierarchy"
        Searcher --> MemoryFirst[Priority 1: Memory Tools]
        MemoryFirst --> AnswerSearch[Search Answers]
        MemoryFirst --> VectorDB

        Searcher --> WebSearch[web_search Orchestrator<br/>picks engine + falls back]
        WebSearch --> ModeLinks[mode: links<br/>Google → DuckDuckGo → Searxng → Firecrawl]
        WebSearch --> ModeAnswer[mode: answer<br/>Tavily → Traversaal → Perplexity → Internal]
        WebSearch --> ModeResearch[mode: research<br/>Perplexity → Tavily → Traversaal → Internal]
        WebSearch --> ModeExploit[mode: exploit<br/>Sploitus → Tavily → Traversaal → Perplexity]

        Searcher --> Browser[browser: read a specific URL]
    end
    
    subgraph "Adviser Workflows"
        Adviser[Adviser Agent<br/>Technical Solution Expert]
        Adviser --> Enricher[Enricher Agent<br/>Context Enhancement]
        Enricher --> Memorist
        Enricher --> Searcher
        
        AdviserNote["Also used for:<br/>- Mentor (execution monitoring)<br/>- Planner (task planning)"]
    end
    
    subgraph "Error Correction"
        Reflector[Reflector Agent<br/>Unstructured Response Corrector]
        PA -.->|No tool calls| Reflector
        Reflector -.->|Corrected instruction| PA
    end
    
    subgraph "Barrier Functions"
        Done[done Tool<br/>Complete Subtask]
        Ask[ask Tool<br/>Request User Input]
    end
    
    PA --> Done
    PA --> Ask
    
    subgraph "Execution Environment"
        Terminal --> DockerContainer[Docker Container<br/>Isolated Environment]
        File --> DockerContainer
        Browser --> WebScraper[Web Scraper Container]
    end
    
    subgraph "Vector Storage Types"
        VectorDB --> GuideStore[Guide Storage<br/>doc_type: guide]
        VectorDB --> AnswerStore[Answer Storage<br/>doc_type: answer]
        VectorDB --> CodeStore[Code Storage<br/>doc_type: code]
        VectorDB --> MemoryStore[Memory Storage<br/>doc_type: memory]
    end
```

## 4. Supporting Workflows

### Assistant Mode Workflow

```mermaid
sequenceDiagram
    participant U as User
    participant AW as AssistantWorker
    participant AP as AssistantProvider
    participant AA as Assistant Agent
    participant Specialists as Specialist Agents
    participant DirectTools as Direct Tools
    participant Stream as Message Stream
    
    U->>AW: Interactive request with UseAgents flag
    AW->>AP: Process input
    AP->>AA: Execute with UseAgents configuration
    
    alt UseAgents = true
        Note over AA,Specialists: Full agent delegation enabled
        AA->>Specialists: search, pentester, coder, advice, memorist, maintenance
        Specialists-->>AA: Structured agent responses
        AA->>DirectTools: terminal, file, browser
        DirectTools-->>AA: Direct tool responses
    else UseAgents = false
        Note over AA,DirectTools: Direct tools only mode  
        AA->>DirectTools: terminal, file, browser
        AA->>DirectTools: web_search (modes: links/answer/research/exploit)
        AA->>DirectTools: search_in_memory, search_guide, search_answer, search_code
        DirectTools-->>AA: Tool responses (no agent delegation)
    end
    
    AA->>Stream: Stream response chunks (thinking/content/updates)
    Stream-->>U: Real-time streaming updates
    
    opt Conversation continues
        U->>AW: Follow-up input
        Note over AW: Message chain preserved in DB
        AW->>AP: Continue conversation context
    end
    
    opt FlowWorker injected (flow management enabled)
        Note over AP,AA: get_flow_status / stop_flow / submit_flow_input / patch_flow_subtasks available
        AA->>AP: Call flow management tool
        AP->>AP: buildFlowManagerHandlers() → stopAssistantFlow / sendAssistantFlowInput / patchAssistantFlowSubtasks
        AP-->>AA: Tool result with actual flow state
    end
```

### Vector Database (RAG) Integration

```mermaid
graph LR
    subgraph "Knowledge Storage Types"
        Guides[Installation Guides<br/>doc_type: guide<br/>guide_type: install/configure/use/etc]
        Answers[Q&A Pairs<br/>doc_type: answer<br/>answer_type: guide/vulnerability/code/tool/other]
        Code[Code Samples<br/>doc_type: code<br/>code_lang: python/bash/etc]
        Memory[Execution Memory<br/>doc_type: memory<br/>tool_name + results]
    end
    
    subgraph "Vector Operations (threshold: 0.2, limit: 3)"
        SearchOps[search_guide<br/>search_answer<br/>search_code<br/>search_in_memory]
        StoreOps[store_guide<br/>store_answer<br/>store_code<br/>auto-store from 18 tools]
    end
    
    subgraph "Auto-Storage Tools"
        EnvTools[terminal, file]
        SearchEngines[web_search]
        AgentTools[search, maintenance, coder,<br/>pentester, advice]
    end
    
    SearchOps --> Guides
    SearchOps --> Answers  
    SearchOps --> Code
    SearchOps --> Memory
    
    StoreOps --> Guides
    StoreOps --> Answers
    StoreOps --> Code
    StoreOps --> Memory
    
    EnvTools --> Memory
    SearchEngines --> Memory
    AgentTools --> Memory
    
    subgraph "Vector Database"
        PostgreSQL[(PostgreSQL + pgvector<br/>Similarity Search<br/>Metadata Filtering)]
    end
    
    SearchOps --> PostgreSQL
    StoreOps --> PostgreSQL
    Memory --> PostgreSQL
```

### Multi-Provider LLM Integration

```mermaid
graph TD
    PC[ProviderController] --> OpenAI[OpenAI Provider]
    PC --> Anthropic[Anthropic Provider]
    PC --> Gemini[Gemini Provider]
    PC --> Bedrock[AWS Bedrock Provider]
    PC --> DeepSeek[DeepSeek Provider]
    PC --> GLM[Zhipu AI Provider]
    PC --> Kimi[Moonshot AI Provider]
    PC --> Qwen[Alibaba Cloud DashScope Provider]
    PC --> Ollama[Ollama Provider]
    PC --> Custom[Custom Provider]
    
    subgraph "Agent Configurations"
        Simple[Simple Agent]
        JSON[Simple JSON Agent]
        Primary[Primary Agent]
        Assistant[Assistant Agent]
        Generator[Generator Agent]
        Refiner[Refiner Agent]
        Adviser[Adviser Agent]
        Reflector[Reflector Agent]
        Searcher[Searcher Agent]
        Enricher[Enricher Agent]
        Coder[Coder Agent]
        Installer[Installer Agent]
        Pentester[Pentester Agent]
    end
    
    OpenAI --> Simple
    OpenAI --> JSON
    OpenAI --> Primary
    OpenAI --> Assistant
    OpenAI --> Generator
    OpenAI --> Refiner
    OpenAI --> Adviser
    OpenAI --> Reflector
    OpenAI --> Searcher
    OpenAI --> Enricher
    OpenAI --> Coder
    OpenAI --> Installer
    OpenAI --> Pentester
    
    Note1[Each provider supports 13 agent types:<br/>Simple, SimpleJSON, PrimaryAgent, Assistant<br/>Generator, Refiner, Adviser, Reflector<br/>Searcher, Enricher, Coder, Installer, Pentester]
```

### Tool Execution and Context Management

```mermaid
graph TD
    Agent[AI Agent] --> ContextSetup[Set Agent Context<br/>ParentAgent → CurrentAgent]
    ContextSetup --> ToolCall[Tool Call Execution]
    
    ToolCall --> Logging[Tool Call Logging<br/>Store in database]
    Logging --> MessageLog[Message Log Creation<br/>Thinking + Message]
    MessageLog --> Execution[Execute Handler]
    
    Execution --> Success{Execution<br/>Successful?}
    Success -->|Yes| StoreMemory[Store in Vector DB<br/>If allowed tool type]
    StoreMemory --> UpdateResult[Update Message Result]
    UpdateResult --> Continue[Continue Workflow]
    
    Success -->|No| ErrorType{Error Type?}
    
    ErrorType -->|Invalid JSON| ToolCallFixer[Tool Call Fixer Agent]
    ToolCallFixer --> FixedJSON[Corrected JSON Arguments]
    FixedJSON --> Retry1[Retry Execution]
    
    ErrorType -->|Other Error| Retry2[Direct Retry]
    Retry1 --> RetryCount{Retry Count<br/>< 3?}
    Retry2 --> RetryCount
    
    RetryCount -->|Yes| ToolCall
    RetryCount -->|No| RepeatingDetector[Repeating Detector]
    
    RepeatingDetector --> BlockTool[Block Tool Call]
    BlockTool --> Agent
    
    Agent --> NoToolCalls{Returns<br/>Unstructured Text?}
    NoToolCalls -->|Yes| Reflector[Reflector Agent]
    Reflector --> UserGuidance[User-style Guidance]
    UserGuidance --> Agent
    
    NoToolCalls -->|No| Continue
    
    subgraph "Memory Storage Rules"
        AllowedTools[18 Allowed Tools:<br/>terminal, file, search engines,<br/>agent delegation tools]
        AutoSummarize[Auto Summarize:<br/>terminal, browser > 16KB]
    end
    
    StoreMemory --> AllowedTools
    UpdateResult --> AutoSummarize
```

### Comprehensive Logging Architecture

```mermaid
graph TB
    subgraph "Flow Execution Hierarchy"
        Flow[Flow Worker] --> Task[Task Worker] --> Subtask[Subtask Worker]
        Flow --> Assistant[Assistant Worker]
    end
    
    subgraph "Controller Layer"
        FlowCtrl[Flow Controller]
        MsgLogCtrl[Message Log Controller]
        AgentLogCtrl[Agent Log Controller]  
        SearchLogCtrl[Search Log Controller]
        TermLogCtrl[Terminal Log Controller]
        VectorLogCtrl[Vector Store Log Controller]
        ToolCallLogCtrl[Tool Call Log Controller]
        ScreenshotCtrl[Screenshot Controller]
        AssistantLogCtrl[Assistant Log Controller]
    end
    
    subgraph "Worker Layer (per Flow)"
        MsgLogWorker[Flow Message Log Worker]
        AgentLogWorker[Flow Agent Log Worker]
        SearchLogWorker[Flow Search Log Worker]  
        TermLogWorker[Flow Terminal Log Worker]
        VectorLogWorker[Flow Vector Store Log Worker]
        ToolCallLogWorker[Flow Tool Call Log Worker]
        ScreenshotWorker[Flow Screenshot Worker]
        AssistantLogWorker[Flow Assistant Log Worker]
    end
    
    subgraph "Database Logging"
        MsgLogDB[(Message Logs<br/>User interactions)]
        AgentLogDB[(Agent Logs<br/>Initiator → Executor)]
        SearchLogDB[(Search Logs<br/>Engine + Query + Result)]
        TermLogDB[(Terminal Logs<br/>Stdin/Stdout)]
        VectorLogDB[(Vector Store Logs<br/>Retrieve/Store actions)]
        ToolCallLogDB[(Tool Call Logs<br/>received/running/finished/failed)]
        ScreenshotDB[(Screenshots<br/>Browser captures)]
        AssistantLogDB[(Assistant Logs<br/>Interactive conversation)]
    end
    
    subgraph "Real-time Updates"
        GraphQL[GraphQL Subscriptions]
        Publisher[Flow Publisher]
    end
    
    FlowCtrl --> MsgLogCtrl
    FlowCtrl --> AgentLogCtrl
    FlowCtrl --> SearchLogCtrl
    FlowCtrl --> TermLogCtrl
    FlowCtrl --> VectorLogCtrl
    FlowCtrl --> ToolCallLogCtrl
    FlowCtrl --> ScreenshotCtrl
    FlowCtrl --> AssistantLogCtrl
    
    MsgLogCtrl --> MsgLogWorker
    AgentLogCtrl --> AgentLogWorker
    SearchLogCtrl --> SearchLogWorker
    TermLogCtrl --> TermLogWorker
    VectorLogCtrl --> VectorLogWorker
    ToolCallLogCtrl --> ToolCallLogWorker
    ScreenshotCtrl --> ScreenshotWorker
    AssistantLogCtrl --> AssistantLogWorker
    
    MsgLogWorker --> MsgLogDB
    AgentLogWorker --> AgentLogDB
    SearchLogWorker --> SearchLogDB
    TermLogWorker --> TermLogDB
    VectorLogWorker --> VectorLogDB
    ToolCallLogWorker --> ToolCallLogDB
    ScreenshotWorker --> ScreenshotDB
    AssistantLogWorker --> AssistantLogDB
    
    Flow --> MsgLogWorker
    Subtask --> AgentLogWorker
    Subtask --> SearchLogWorker
    Subtask --> TermLogWorker
    Subtask --> VectorLogWorker
    Subtask --> ToolCallLogWorker
    Subtask --> ScreenshotWorker
    Assistant --> AssistantLogWorker
    
    MsgLogWorker --> Publisher
    AgentLogWorker --> Publisher
    SearchLogWorker --> Publisher
    TermLogWorker --> Publisher
    VectorLogWorker --> Publisher
    ToolCallLogWorker --> Publisher
    ScreenshotWorker --> Publisher
    AssistantLogWorker --> Publisher
    
    Publisher --> GraphQL
```

### Docker Container Management

```mermaid
graph TB
    subgraph "Flow Initialization"
        ImageSelection[Image Chooser Agent<br/>Select optimal image]
        ContainerSpawn[Container Creation<br/>With security capabilities]
    end
    
    subgraph "Container Configuration"
        Primary[Primary Container<br/>Main execution environment]
        Ports[Dynamic Port Allocation<br/>Base: 28000 + flowID*2]
        Volumes[Volume Management<br/>Data persistence]
        Network[Network Configuration<br/>Optional custom network]
    end
    
    subgraph "Tool Execution Environment"
        WorkDir[Work Directory<br/>/work in container]
        Uploads[User Uploads<br/>/work/uploads]
        Resources[User Resources<br/>/work/resources]
        Terminal[Terminal Tool<br/>default 1200s, hard limit 3h]
        FileOps[File Operations<br/>Absolute paths required]
        WebAccess[Web Access<br/>Separate scraper container]
    end
    
    ImageSelection --> Primary
    ContainerSpawn --> Primary
    Primary --> Ports
    Primary --> Volumes  
    Primary --> Network
    Primary --> WorkDir
    WorkDir --> Uploads
    WorkDir --> Resources
    WorkDir --> Terminal
    WorkDir --> FileOps
    Primary --> WebAccess
    
    subgraph "Security Capabilities"
        NetRaw[NET_RAW Capability<br/>Network packet access]
        NetAdmin[NET_ADMIN Capability<br/>Optional: Network admin]
        Isolation[Container Isolation<br/>No host access]
        RestartPolicy[Restart Policy<br/>unless-stopped]
    end
    
    Primary --> NetRaw
    Primary --> NetAdmin
    Primary --> Isolation
    Primary --> RestartPolicy
```

## 5. Complex Interaction Patterns

### Message Chain Management
Each AI agent interaction is managed through typed message chains that maintain conversation context:

**Chain Types by Agent**:
- `MsgchainTypePrimaryAgent` - Primary Agent orchestration chains
- `MsgchainTypeGenerator` - Subtask generation chains  
- `MsgchainTypeRefiner` - Subtask refinement chains
- `MsgchainTypeReporter` - Final report generation chains
- `MsgchainTypeCoder` - Code development chains
- `MsgchainTypePentester` - Security testing chains
- `MsgchainTypeInstaller` - Infrastructure maintenance chains
- `MsgchainTypeMemorist` - Memory operation chains
- `MsgchainTypeSearcher` - Information retrieval chains
- `MsgchainTypeAdviser` - Expert consultation chains
- `MsgchainTypeReflector` - Response correction chains
- `MsgchainTypeAssistant` - Interactive assistance chains
- `MsgchainTypeSummarizer` - Context summarization chains
- `MsgchainTypeToolCallFixer` - Tool argument repair chains

**Chain Properties**:
- **Serialized to JSON** and stored in the database for persistence
- **Summarized periodically** to prevent context window overflow
- **Restored on system restart** to maintain continuity
- **Type-specific retrieval** for agent-specific context loading

### Agent Context Tracking
The system maintains agent execution context through the call chain:

**Agent Context Structure**:
- **ParentAgentType** - The agent that initiated the current operation
- **CurrentAgentType** - The agent currently executing  

**Context Propagation**:
- Set via `PutAgentContext(ctx, agentType)` when invoking agents
- Retrieved via `GetAgentContext(ctx)` for logging and tracing
- Used for vector store logging to track agent delegation chains
- Enables observability of inter-agent communication patterns

**Message Chain Types** (tracks agent interactions):
- `MsgchainTypePrimaryAgent`, `MsgchainTypeGenerator`, `MsgchainTypeRefiner`
- `MsgchainTypeReporter`, `MsgchainTypeCoder`, `MsgchainTypePentester` 
- `MsgchainTypeInstaller`, `MsgchainTypeMemorist`, `MsgchainTypeSearcher`
- `MsgchainTypeAdviser`, `MsgchainTypeReflector`, `MsgchainTypeEnricher`
- `MsgchainTypeAssistant`, `MsgchainTypeSummarizer`, `MsgchainTypeToolCallFixer`

### Agent Chain Execution Loop
The Primary Agent follows a sophisticated execution pattern:

1. **Context Preparation** - Loads execution context including completed/planned Subtasks
2. **Tool Call Loop** - Iteratively calls LLM with available tools until completion
3. **Function Execution** - Executes tool calls with retry logic and error handling
4. **Repeating Detection** - Prevents infinite loops by detecting repeated tool calls (threshold: 3)
5. **Reflection Mechanism** - If no tools are called, invokes Reflector Agent for guidance
6. **Barrier Functions** - Special tools (`done`, `ask`) that control execution flow

### Reflector Agent Correction Mechanism
A critical system component that handles agent errors:

- **Triggers when** - Any agent returns unstructured text instead of structured tool calls
- **Maximum iterations** - Limited to 3 reflector calls per chain to prevent loops
- **Response style** - Acts as the user providing direct, concise guidance
- **Correction process** - Analyzes the unstructured response and guides agent to proper tool usage
- **Barrier tool emphasis** - Specifically reminds agents about completion tools (`done`, `ask`)
- **Assistant exception** - Assistant agents return natural text responses to users, not tool calls
- **Final response mode** - Assistants use completion mode for user-facing communication
- **Context isolation** - Assistants use `nil` taskID/subtaskID when accessing agent handlers
- **Cross-flow operation** - Assistants can access Flow-level context without specific Task/Subtask binding

### Subtask Lifecycle States
Subtasks progress through well-defined states:
- **Created** - Initially generated by Generator Agent
- **Running** - Currently being executed by Primary Agent
- **Waiting** - Paused for user input via `ask` tool
- **Finished** - Successfully completed via `done` tool
- **Failed** - Terminated due to errors

### Error Handling and Recovery
The system implements comprehensive error handling:

**Multi-layer Error Correction**:
- **Tool Call Retries** - Failed LLM calls retried up to 3 times with 5-second delays
- **Tool Call Fixer** - Invalid JSON arguments automatically corrected using schema validation
- **Reflector Correction** - Unstructured responses redirected to proper tool call format
- **Chain Consistency** - Adds default responses to incomplete tool calls after interruptions
- **Repeating Detection** - Prevents infinite loops by limiting identical tool calls to 3 attempts

**Chain Consistency Mechanism**:
- **Triggered on errors** - System interruptions or context cancellations
- **Fallback responses** - Adds default content for unresponded tool calls
- **Message integrity** - Maintains valid conversation structure
- **AST-based processing** - Uses Chain AST for structured message analysis

**System Resilience**:
- **Graceful Degradation** - Falls back to simpler operations when complex ones fail
- **Context Preservation** - Maintains state across system restarts
- **Container cleanup** - Automatic resource cleanup on failures
- **Flow status management** - Proper state transitions on errors

### Task Refinement Process
The Refiner Agent uses a sophisticated analysis approach:
1. **Reviews completed Subtasks** and their execution results
2. **Analyzes remaining planned Subtasks** for relevance
3. **Considers overall Task progress** and user requirements
4. **Updates the Subtask plan** by removing obsolete tasks and adding necessary ones
5. **Maintains execution efficiency** by limiting total Subtasks to 15 maximum
6. **Dynamic limit calculation** - Available slots = 15 minus completed Subtasks count
7. **Completion detection** - Returns empty list when Task objectives are achieved

### Memory and Knowledge Management
The system maintains multiple types of persistent knowledge with PostgreSQL + pgvector:

**Vector Store Types**:
- **Memory Storage** (`doc_type: memory`) - Tool execution results and agent observations
- **Guide Storage** (`doc_type: guide`) - Installation and configuration procedures
- **Answer Storage** (`doc_type: answer`) - Q&A pairs for common scenarios  
- **Code Storage** (`doc_type: code`) - Programming language-specific code samples

**Lifecycle Guidance**:
- Treat `memory` as flow-scoped execution history. It is most useful for understanding what happened in a specific engagement and is commonly inspected with a `flow_id` filter.
- Treat `guide`, `answer`, and `code` as reusable knowledge. These document types exist to preserve durable procedures, reusable target notes, Q&A material, and code snippets across future runs.
- If you want a later flow to begin with known context, store the confirmed result intentionally through `store_guide`, `store_answer`, or `store_code` instead of assuming execution history alone will provide the right reusable context.
- Current prompt templates already distinguish these roles: reusable guides, answers, and code live in vector documents, while Graphiti is intended for episodic memory about what actually happened during execution.

**Technical Parameters**:
- **Similarity Threshold**: 0.2 for all vector searches
- **Result Limits**: 3 documents maximum per search
- **Memory Storage Tools**: 18 tools automatically store results (terminal, file, all search engines, all agent tools)
- **Summarization Eligible**: Only `terminal` and `browser` tools results are auto-summarized when > 16KB

### Search Tool Priority System
The Searcher Agent follows a strict hierarchy for information retrieval:

1. **Priority 1-2: Memory Tools** - Always check internal knowledge first
   - `search_answer` - Primary tool for accessing existing knowledge
   - `memorist` - Retrieves task/subtask execution history

2. **Priority 3: `web_search`** - All external web search, via one unified tool
   - The agent supplies a `query` and an intent `mode`; it does NOT choose a provider.
   - **Modes**: `links` (fast link discovery), `answer` (synthesized answer, default),
     `research` (deep multi-source synthesis), `exploit` (exploits/PoCs/offensive tools).

3. **Priority 4: `browser`** - Targeted content extraction from a specific, known URL

**web_search orchestration**: For the chosen mode, the orchestrator walks a per-mode
fallback chain, skipping engines that are not configured, retrying only *retryable*
failures (HTTP 429/5xx/transport) on the same engine, and moving on immediately for
*fatal* ones (HTTP 4xx/decode). It returns an error to the agent only after every
available engine is exhausted, and it records one search-log entry attributed to the
engine that actually served the result. The fallback table is the single source of
truth at the top of `backend/pkg/tools/web_search.go`.

**Engines orchestrated** (used only when configured):
- **Google** - Custom Search API with CX key and language restrictions (links)
- **DuckDuckGo** - Anonymous search (links)
- **Searxng** - Privacy meta search aggregating multiple engines (links)
- **Firecrawl** - Web search with main-content markdown scraped from each result
- **Tavily** - Research-grade exploration with raw content and citations (answer/research)
- **Traversaal** - Structured Q&A responses with web links (answer/research)
- **Perplexity** - AI-powered synthesis with configurable context size (answer/research)
- **Sploitus** - Security exploits and pentest tools (exploit)
- **Internal Analytics Engine** - optional, off by default (`WEB_SEARCH_INTERNAL_ENABLED`);
  answers analytic queries by scraping and summarizing pages via the browser, bounded to
  a configurable byte budget per site

**Action Economy Rules**: Maximum 3-5 search actions per query, stop immediately when sufficient information is found

### Summarization Protocol
A critical system-wide mechanism for context management:

- **Two Summary Types**:
  1. **Tool Call Summary** - AI message with only `SummarizationToolName` tool call
  2. **Prefixed Summary** - AI message starting with `SummarizedContentPrefix`
  
- **Agent Handling Rules**:
  - Must treat summaries as **historical records** of actual past events
  - Extract useful information to inform current strategy
  - **Never mimic** summary formats or use summarization tools
  - Continue using structured tool calls for all actions
  
- **System Benefits**:
  - Prevents context window overflow during long conversations
  - Maintains conversation coherence across system restarts
  - Preserves critical execution context while reducing token usage

### Real-time Communication System

```mermaid
graph LR
    subgraph "Stream Processing"
        Agent[AI Agent] --> StreamID[Generate Stream ID]
        StreamID --> ThinkingChunk[Thinking Chunks<br/>Reasoning Process]
        StreamID --> ContentChunk[Content Chunks<br/>Incremental Building]
        StreamID --> UpdateChunk[Update Chunks<br/>Complete Sections]
        StreamID --> FlushChunk[Flush Chunks<br/>Segment Completion]
        StreamID --> ResultChunk[Result Chunks<br/>Final Results]
    end
    
    subgraph "Assistant Streaming"
        AssistantAgent[Assistant Agent] --> StreamCache[Stream Cache<br/>LRU 1000 entries, 2h TTL]
        StreamCache --> StreamWorker[Stream Worker<br/>30s timeout]
        StreamWorker --> AssistantUpdate[Real-time Updates]
    end
    
    subgraph "Real-time Distribution"
        Publisher[Flow Publisher] --> GraphQLSubs[GraphQL Subscriptions]
        GraphQLSubs --> FlowCreated[Flow Created/Updated]
        GraphQLSubs --> TaskCreated[Task Created/Updated] 
        GraphQLSubs --> AgentLogAdded[Agent Log Added]
        GraphQLSubs --> MessageLogAdded[Message Log Added/Updated]
        GraphQLSubs --> TerminalLogAdded[Terminal Log Added]
        GraphQLSubs --> SearchLogAdded[Search Log Added]
        GraphQLSubs --> VectorStoreLogAdded[Vector Store Log Added]
        GraphQLSubs --> ToolCallLogAdded[Tool Call Log Added/Updated]
        GraphQLSubs --> ScreenshotAdded[Screenshot Added]
        GraphQLSubs --> AssistantLogAdded[Assistant Log Added/Updated]
    end
    
    ThinkingChunk --> Publisher
    ContentChunk --> Publisher  
    UpdateChunk --> Publisher
    FlushChunk --> Publisher
    ResultChunk --> Publisher
    AssistantUpdate --> Publisher
```

### Multi-tenancy and Security
The architecture supports multiple users with isolation:
- **User-specific providers** - Each user can configure their own LLM providers
- **Flow isolation** - Docker containers provide security boundaries
- **Resource management** - Containers are automatically cleaned up
- **Access control** - Database queries are user-scoped

### Specialized System Prompts
The system uses 25+ dedicated prompt types for specific functions:

**System Function Prompts**:
- **Image Chooser** - Selects optimal Docker image, fallback to `vxcontrol/kali-linux` for pentest
- **Language Chooser** - Detects user's preferred language for responses  
- **Flow Descriptor** - Generates concise Flow titles (max 20 characters)
- **Task Descriptor** - Creates descriptive Task titles (max 150 characters)  
- **Tool Call Fixer** - Repairs invalid JSON arguments using schema validation
- **Execution Context** - Templates for full/short context summaries
- **Execution Logs** - Formats chronological action histories for summarization

**Agent System Prompts** (13 types):
- **Primary Agent** - Team orchestration with delegation capabilities
- **Assistant** - Interactive mode with UseAgents flag configuration
- **Specialist Agents** - Pentester, Coder, Installer, Searcher, Memorist, Adviser
- **Meta Agents** - Generator, Refiner, Reporter, Enricher, Reflector, Summarizer

**Question Templates** (13 types):
- Structured input templates for each agent's human interaction patterns
- Context-specific variable injection for Task/Subtask/Flow information
- Formatted data presentation for optimal agent comprehension

**Critical Prompt Features**:
- **XML Semantic Delimiters** - Structured sections like `<memory_protocol>`, `<terminal_protocol>`
- **Summarization Awareness** - Universal protocol for handling historical summaries  
- **Tool Placeholder System** - `{{.ToolPlaceholder}}` injection at prompt end
- **Task Materials Section** - `{{.UserFiles}}` lists user uploads/resources when present
- **Template Variable System** - 50+ variables for dynamic content injection

### Performance Optimizations and Limits
Several mechanisms ensure efficient execution:

**Execution Limits**:
- **Subtask Limits** - Maximum 15 Subtasks per Task (TasksNumberLimit)
- **Refiner Calculations** - Available slots = 15 minus completed Subtasks count
- **Reflector Iterations** - Maximum 3 corrections per agent chain
- **Tool Call Retries** - Maximum 3 attempts for failed executions
- **Repeating Detection** - Blocks repeated tool calls after 3 attempts
- **Search Action Economy** - Searcher limited to 3-5 actions per query

**Timeout Configuration**:
- **Terminal Operations** - Configurable via `TERMINAL_TOOL_TIMEOUT` (default: 1200s); hard limit 3h/10800s; 0 or negative values clamped to hard limit
- **LLM API Calls** - 3 retries with 5-second delays between attempts  
- **Vector Search** - Threshold 0.2, max 3 results per query
- **Tool Result Summarization** - Triggered at 16KB result size
- **Flow Input Processing** - 1 second timeout for input queueing
- **Assistant Input** - 2 second timeout for assistant input queueing
- **Flow Management Operations** - 15 second timeout for `stop_flow` and `submit_flow_input` handlers

**Container Resource Management**:
- **Port Allocation** - 2 ports per Flow starting from base 28000
- **Volume Management** - Per-flow data directories with cleanup
- **File Sync** - Incremental sync of `uploads/` and `resources/` into `/work/uploads` and `/work/resources`
- **Network Isolation** - Optional custom Docker networks
- **Image Fallback** - Automatic fallback to default Debian image

**Memory Optimization**:
- **Message summarization** - Prevents context window overflow
- **Summarizer result cache** - LRU cache per FlowProvider (1000 entries, 4h TTL); keyed by SHA-256 of input; skips LLM call for repeated identical content
- **Tool argument limits** - 1KB limit for individual argument values  
- **Connection pooling** - Database connections are reused
- **Automatic cleanup** - Containers removed after Flow completion

### User Interaction Flow (`ask` Tool)
Critical mechanism for human-in-the-loop operations:

**Ask User Workflow**:
1. **Primary Agent** calls `ask` tool with question for user
2. **PerformResultWaiting** returned to SubtaskWorker
3. **Subtask status** set to `SubtaskStatusWaiting`  
4. **Task status** propagated to `TaskStatusWaiting`
5. **Flow status** propagated to `FlowStatusWaiting`
6. **User provides input** via Flow interface
7. **Input processed** through `updateMsgChainResult` with `AskUserToolName`
8. **Subtask continues** execution with user's response

**Configuration**:
- **ASK_USER environment variable** - Controls availability of `ask` tool
- **Default**: false (disabled by default)
- **Primary Agent only** - Only available in PrimaryExecutor configuration
- **Barrier function** - Causes execution flow pause until user responds

### Browser and Screenshot System
The browser tool provides advanced web interaction capabilities:

**Browser Actions**:
- **Markdown Extraction** - Clean text content from web pages  
- **HTML Content** - Raw HTML for detailed analysis
- **Link Extraction** - Collect all URLs from pages for further navigation

**Screenshot Integration**:
- **Automatic Screenshots** - Every browser action captures page screenshot
- **Dual Scraper Support** - Private URL scraper for internal networks, public for external
- **Screenshot Storage** - Organized by Flow ID with timestamp naming
- **Minimum Content Sizes** - MD: 50 bytes, HTML: 300 bytes, Images: 2048 bytes

**Network Resolution**:
- **IP Analysis** - Automatic detection of private vs public targets
- **Scraper Selection** - Private scraper for internal IPs, public for external
- **Security Isolation** - Web scraping isolated from main execution container

## Advanced Agent Supervision

PentAGI implements a sophisticated multi-layered agent supervision system to ensure efficient task execution, prevent infinite loops, and provide intelligent recovery from stuck states.

### Execution Monitoring System

**ExecutionMonitorDetector** continuously monitors agent tool call patterns and automatically invokes the Adviser agent for progress reviews:

**Trigger Conditions**:
- **Same Tool Threshold**: Triggered after 5 consecutive calls to the same tool (configurable via `EXECUTION_MONITOR_SAME_TOOL_LIMIT`)
- **Total Tool Threshold**: Triggered after 10 total tool calls regardless of variety (configurable via `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT`)
- **Reset Behavior**: Counters reset after adviser intervention or when different tools are used

**Monitoring Process**:
1. **Pattern Detection**: `execToolCall` method checks detector before executing each tool
2. **Context Collection**: Gathers recent messages, executed tool calls, subtask description, and agent prompt
3. **Mentor Invocation**: Calls `performMentor` with comprehensive execution context
4. **Enhanced Response**: Mentor analysis is formatted as `<mentor_analysis>` alongside `<original_result>`
5. **Counter Reset**: Monitor state resets after successful intervention

**Mentor Analysis Provides**:
- **Progress Assessment**: Evaluation of whether agent is advancing toward subtask objective
- **Issue Identification**: Detection of loops, inefficiencies, or incorrect approaches
- **Alternative Strategies**: Recommendations for different approaches when current strategy fails
- **Information Retrieval Guidance**: Suggestions to search for established solutions instead of reinventing
- **Termination Guidance**: Clear indication if task is impossible or should be completed with completion function call

**Configuration**:
- `EXECUTION_MONITOR_ENABLED` (default: false) - Enable/disable automatic monitoring
- `EXECUTION_MONITOR_SAME_TOOL_LIMIT` (default: 5) - Consecutive same-tool threshold
- `EXECUTION_MONITOR_TOTAL_TOOL_LIMIT` (default: 10) - Total tool calls threshold

For local or custom Qwen-style deployments where flows run for hours or repeat commands, see the [vLLM Qwen3.5-27B-FP8 troubleshooting guide](../../examples/guides/vllm-qwen35-27b-fp8.md#issue-automation-flow-runs-for-hours-or-repeats-commands) for prompt-bounding and log-review guidance.

### Enhanced Reflector Integration

**Automatic Reflector on Generation Failures**:

When LLM fails to generate valid tool calls after 3 attempts in `callWithRetries`, the system now automatically invokes the Reflector agent instead of failing:

**Invocation Process**:
1. **Failure Detection**: `callWithRetries` reaches `maxRetriesToCallAgentChain` (3 attempts)
2. **Context Preparation**: Builds reflector message describing all failed attempts and errors
3. **Reflector Call**: Invokes `performReflector` to analyze situation and provide guidance
4. **Recovery Options**: Reflector guides agent to either:
   - Fix the issue with specific corrective instructions
   - Use barrier tool to report completion or request assistance

**Benefits**:
- Prevents premature task termination due to transient LLM issues
- Provides contextual guidance based on specific failure patterns
- Maintains conversation flow rather than hard errors
- Enables graceful degradation and adaptive recovery

### Hard Limit Graceful Termination

**Max Tool Calls Per Agent Execution**:

To prevent runaway executions, each agent has a hard limit on tool calls. The limit varies by agent type to balance capabilities with efficiency:

**Agent Types and Limits**:
- **General Agents** (Assistant, Primary Agent, Pentester, Coder, Installer):
  - Default: 100 tool calls
  - Configurable via `MAX_GENERAL_AGENT_TOOL_CALLS`
  - Designed for complex, multi-step workflows requiring extensive tool usage
  
- **Limited Agents** (Searcher, Enricher, Memorist, Generator, Reporter, Adviser, Reflector, Planner):
  - Default: 20 tool calls
  - Configurable via `MAX_LIMITED_AGENT_TOOL_CALLS`
  - Designed for focused, specific tasks with limited scope

**Termination Process**:
1. **Limit Check**: Before each `callWithRetries` in `performAgentChain`, system checks `iteration` against agent-specific limit
2. **Reflector Invocation**: When approaching limit (within 3 iterations), reflector is called with termination context
3. **Graceful Completion**: Reflector guides agent to use barrier tool (`done` or `ask`) to:
   - Report successful completion if objective was achieved
   - Report partial progress with clear blocker explanation
   - Request user assistance if critical information is missing
4. **Forced Exit**: After reflector guidance, execution terminates gracefully

**Configuration**:
- `MAX_GENERAL_AGENT_TOOL_CALLS` (default: 100) - Maximum tool calls for general agents before forced termination
- `MAX_LIMITED_AGENT_TOOL_CALLS` (default: 20) - Maximum tool calls for limited agents before forced termination

**Why Differentiated Limits**:
- **Resource Efficiency**: Limited agents handle focused tasks and don't require extensive iteration
- **Task Complexity**: General agents need more autonomy for complex penetration testing, coding, and installation workflows
- **System Stability**: Prevents resource exhaustion while maintaining necessary capabilities for each agent type

### Intelligent Task Planning (Planner)

**Planner-Generated Execution Plans**:

When specialist agents (Pentester, Coder, Installer) are invoked, the Planner (adviser in planning mode) optionally generates a structured execution plan before task execution:

**Planning Process**:
1. **Context Analysis**: Planner analyzes full execution context via enricher agent
2. **Plan Generation**: Creates 3-7 specific, actionable steps via `PromptTypeQuestionTaskPlanner` template
3. **Scope Limitation**: Ensures plan focuses only on current subtask objective
4. **Plan Wrapping**: Original task question is wrapped in `<task_assignment>` structure with plan
5. **Agent Execution**: Specialist receives both original request and decomposed execution plan

**Plan Structure**:
```xml
<task_assignment>
  <original_request>[Original task from delegating agent]</original_request>
  <execution_plan>
  1. [First critical action/verification]
  2. [Second step with specific details]
  ...
  </execution_plan>
  <instructions>
  Follow the execution plan above to complete this task efficiently.
  You may deviate from the plan if you discover better approaches.
  </instructions>
</task_assignment>
```

**Benefits**:
- **Prevents scope creep**: Keeps agents focused on current subtask only
- **Reduces redundancy**: Leverages enriched context to avoid duplicate work
- **Improves success rate**: Breaks complex tasks into manageable steps
- **Provides guardrails**: Highlights potential pitfalls and verification points

**Configuration**:
- `AGENT_PLANNING_STEP_ENABLED` (default: false) - Enable/disable automatic task planning

### Mentor Supervision Protocol

All agents with adviser handler access (Primary, Pentester, Coder, Installer, Assistant) now include explicit awareness of mentor supervision in their system prompts:

**Enhanced Response Format**:
Agents are instructed to expect tool responses containing both:
- `<original_result>`: Actual tool execution output
- `<mentor_analysis>`: Mentor's evaluation with progress assessment, identified issues, alternative approaches, information retrieval suggestions, and next steps

**Agent Instructions**:
- Agents must read and integrate BOTH sections into decision-making
- Mentor analysis should guide next actions when provided
- Agents can explicitly request advice via `advice` tool
- Automatic mentor reviews occur at configured thresholds (not revealed to agents)

### Supervision System Integration

```mermaid
graph TB
    subgraph "Execution Monitoring (Mentor)"
        ToolCall[Tool Call Execution]
        EMD[ExecutionMonitorDetector]
        MentorCheck{Threshold<br/>Reached?}
        InvokeMentor[performMentor]
        Analysis[Mentor Analysis]
        EnhancedResp[Enhanced Response]
    end
    
    subgraph "Generation Failure Recovery"
        CallRetries[callWithRetries Loop]
        MaxRetries{Max Retries<br/>Reached?}
        InvokeReflector1[Invoke Reflector]
        Guidance[Corrective Guidance]
    end
    
    subgraph "Hard Limit Termination"
        AgentChain[performAgentChain Loop]
        LimitCheck{Tool Call<br/>Limit?}
        InvokeReflector2[Invoke Reflector]
        GracefulExit[Graceful Termination]
    end
    
    subgraph "Task Planning (Planner)"
        SpecialistStart[Specialist Agent Start]
        PlanCheck{Planning<br/>Enabled?}
        GetPlan[performPlanner]
        WrapPrompt[Wrap with Plan]
        Execute[Execute with Plan]
    end
    
    ToolCall --> EMD
    EMD --> MentorCheck
    MentorCheck -->|Yes| InvokeMentor
    MentorCheck -->|No| Continue[Continue Execution]
    InvokeMentor --> Analysis
    Analysis --> EnhancedResp
    
    CallRetries --> MaxRetries
    MaxRetries -->|Yes| InvokeReflector1
    MaxRetries -->|No| Retry[Retry Call]
    InvokeReflector1 --> Guidance
    
    AgentChain --> LimitCheck
    LimitCheck -->|Limit Exceeded| InvokeReflector2
    LimitCheck -->|Within Limit| Continue
    InvokeReflector2 --> GracefulExit
    
    SpecialistStart --> PlanCheck
    PlanCheck -->|Yes| GetPlan
    PlanCheck -->|No| Execute
    GetPlan --> WrapPrompt
    WrapPrompt --> Execute
```

### Implementation Details

**Key Components**:
- `executionMonitorDetector` struct in `helpers.go` - Tracks tool call patterns
- `performMentor` method in `performer.go` - Coordinates mentor invocation for execution monitoring
- `performPlanner` method in `performers.go` - Generates execution plans via adviser
- `formatEnhancedToolResponse` function in `helpers.go` - Formats mentor analysis
- `flowStatusTool`, `stopFlowTool`, `submitFlowInputTool`, `patchFlowSubtasksTool` in `tools/flow_manager.go` - Flow management tool implementations
- `buildFlowManagerHandlers()` in `providers/assistant.go` - Builds handler closures from injected `FlowWorker`
- `patchAssistantFlowSubtasks()` in `providers/assistant.go` - Applies subtask delta operations and recreates DB entries
- Summarizer LRU cache on `flowProvider` (`providers/provider.go`) - 1000 entries, 4h TTL, SHA-256 key
- Template `question_execution_monitor.tmpl` - Question format for execution monitoring
- Template `question_task_planner.tmpl` - Question format for task planning

**Modified Methods**:
- `execToolCall`: Integrated execution monitor checks before tool execution
- `callWithRetries`: Calls reflector on max retries instead of returning error
- `performAgentChain`: Checks hard limit and invokes reflector for graceful termination
- `performPentester/Coder/Installer`: Apply task planning before execution

**Execution Limits Updated**:
- **Repeating Tool Threshold**: 3 identical calls (existing)
- **Execution Monitor Same Tool**: 5 identical calls (new)
- **Execution Monitor Total Tools**: 10 total calls (new)
- **Max Retries per Call**: 3 attempts (existing)
- **Max Retries per Chain**: 3 attempts → Reflector invocation (modified)
- **Max Tool Calls per Subtask**: 100 calls → Reflector termination (new)
- **Max Reflector Iterations**: 3 per chain (existing)
- **Max Agent Chain Iterations**: 100 total (existing)

## Summary

The Flow execution system represents a sophisticated orchestration platform that combines multiple AI agents, tools, and infrastructure components to deliver autonomous penetration testing capabilities. Key architectural highlights include:

### System Robustness
- **Triple-layer error handling** - Tool call fixing, reflector correction, and retry mechanisms
- **Context preservation** - Typed message chains with summarization for long-running operations
- **Resource limits** - Bounded execution with configurable timeouts and iteration limits
- **Isolation guarantees** - Docker containers provide security boundaries for all operations

### Agent Specialization
- **Role-based delegation** - Each agent has specific expertise and tool access
- **Memory-first approach** - All agents check vector storage before external operations  
- **Structured communication** - Exclusive use of tool calls except Assistant final responses
- **Adaptive planning** - Generator creates initial plans, Refiner optimizes based on results
- **Context propagation** - Parent/Current agent tracking for delegation chains
- **Categorized tools** - 6 tool categories with specific access patterns

### Operational Flexibility
- **Multi-provider LLM support** - Different models optimized for different agent types
- **Assistant dual modes** - UseAgents flag enables delegation or direct tool access; FlowWorker injection enables 4 flow management tools for automation control
- **Flow continuity** - System can resume operations after interruptions
- **Real-time feedback** - Streaming responses provide immediate user visibility
- **Vector knowledge system** - 4 storage types with semantic search and metadata filtering
- **Comprehensive tool ecosystem** - 44+ tools across 7 categories with automatic memory storage
- **GraphQL subscriptions** - Real-time Flow/Task/Log updates via WebSocket connections
- **Logging architecture** - 8-layer logging system with Controller/Worker pattern

### Critical Technical Details

**Assistant Streaming Architecture**:
- **Stream Cache**: LRU cache (1000 entries, 2h TTL) maps StreamID → MessageID
- **Stream Workers**: Background goroutines with 30-second timeout per stream
- **Buffer Management**: Separate buffers for thinking/content with controlled updates
- **Real-time Distribution**: Immediate GraphQL subscription updates to frontend

**Assistant Flow Management Architecture**:
- **FlowWorker injection**: `SetFlowWorker(fw FlowWorker)` called after `getFlowProviderWorkers()` in `controller/assistant.go`
- **Handler closures**: `buildFlowManagerHandlers()` produces `StopFlow`, `SendFlowInput`, `PatchSubtasks` callbacks
- **Two-layer validation**: tool handlers (`tools/flow_manager.go`) perform primary checks; provider callbacks are safety nets
- **Subtask patching**: delta operations applied via `applySubtaskOperations()`; all planned subtask IDs are recreated; new IDs returned in tool result
- **State verification**: `stop_flow` re-queries `GetFlowTasks` after handler returns and reports actual state
- **Resource handoff**: assistant inputs with resource IDs delegate file copying to `FlowWorker.PutResources()`

**Message Chain Consistency**:
- **AST Processing**: Uses Chain Abstract Syntax Tree for structured message analysis
- **Fallback Content**: Unresponded tool calls get default responses via `cast.FallbackResponseContent`
- **Chain Restoration**: Intelligent restoration of conversation context after interruptions  
- **Summarization Integration**: Seamless integration with summarization when restoring chains

**Flow Publisher Integration**:
- **Centralized Updates**: Single publisher coordinates all Flow-related real-time updates
- **Event Types**: 10 different event types (Flow, Task, Agent logs, Message logs, Tool Call logs, etc.)
- **User-scoped**: Each user gets their own publisher instance for proper isolation
- **WebSocket Distribution**: Efficient real-time delivery to frontend clients

### Implementation Architecture Summary

**User Files and Resources**:
- **User Resources** - Per-user blob storage with metadata in `user_resources`
- **Flow Files** - Per-flow filesystem cache under `flow-{id}-data/{uploads,container,resources}`
- **Container Availability** - `FlowToolsExecutor.Prepare()` incrementally syncs missing `uploads/` and `resources/` files into `/work`
- **Prompt Visibility** - Agents receive a compact `<task_files>` listing only when user files are present

**Core Flow Processing**:
- **3-layer hierarchy**: FlowWorker → TaskWorker → SubtaskWorker with proper lifecycle management
- **Agent orchestration**: 13 specialized agent types with role-specific tool access
- **Tool ecosystem**: 44+ tools across 7 categories (Environment, SearchNetwork, SearchVectorDb, Agent, StoreAgentResult, Barrier, FlowManagement)
- **Message chain types**: 14 distinct chain types for agent communication tracking

**Error Resilience & Recovery**:
- **4-level error handling**: Tool retries → Tool call fixing → Reflector correction → Chain consistency
- **Bounded execution**: Configurable limits preventing runaway operations
- **State recovery**: Complete system state restoration after interruptions
- **Graceful degradation**: Automatic fallbacks to simpler operational modes

**Real-time & Observability**:
- **8-layer logging**: Comprehensive tracking from agent interactions to terminal commands
- **GraphQL subscriptions**: Real-time updates via WebSocket connections
- **Streaming architecture**: Progressive response delivery with thinking/content separation
- **Vector observability**: Complete tracking of knowledge storage and retrieval operations

**Security & Isolation**:
- **Container isolation**: Docker-based security boundaries with capability controls
- **Multi-tenant design**: User-scoped operations with resource isolation  
- **Network segmentation**: Separate containers for web scraping vs. tool execution
- **Resource limits**: Comprehensive timeout and resource management

This architecture enables autonomous security testing while maintaining human oversight, technical precision, and operational security throughout the entire penetration testing workflow.