AI Agents Action Selection
Explore AI agents' action selection, integration challenges, and security in automation. Learn about personal AI agents and best practices.
The Real Advance in AI Agents Is Choosing How to Act
The important agent capability is action selection
The striking development in agents is not that they can send a WhatsApp message, write Python, or click a browser button. Each of those abilities has existed in partial form for years. The harder problem is deciding which action is appropriate.
Sam Witteveen’s examination of H Company’s Holo4 model frames the issue well. A real workflow may require extracting a report from an old browser application, cleaning it with code, then posting results into a modern system through an API.
Those are three different operating environments. A graphical interface is built for a person looking at a screen. Code works against files and computable data. APIs and Model Context Protocol, or MCP, tools expose structured actions for software.
The agent should not use them interchangeably. Clicking through five pages to retrieve data is wasteful when a documented API can return it in one call. Conversely, an API-first agent cannot access a legacy application that only exposes buttons and tables.
That choice, rather than fluent chat, is what determines whether automation is robust or merely impressive in a demo. The model proposes an action, but a surrounding software layer, usually called a harness, actually carries it out.
The agent loop is simpler than the product pages imply
An agentic system repeatedly executes a basic loop: inspect the current state, choose one action, run it through the harness, observe the result, and repeat. The model supplies decisions, not direct physical control.
For a browser task, the observation might be a screenshot, page structure, or accessibility tree. For code, it might be terminal output or a test result. For an API, it is generally structured data such as JSON.
This distinction matters because screenshots are expensive context. A browser image consumes substantially more model memory than an API response saying a payment succeeded or a small terminal log reporting a failed test.
Holo4, as described by Sam Witteveen, is an open-weight model trained to select between GUI interaction, code execution, and tool calls. H Company offers a 27-billion-parameter dense model, a 35-billion-parameter mixture-of-experts variant, and a smaller Holotron Nano family member.
The practical claim is not that one model can magically operate every system. It is that a single action model can route work across three lanes without an engineer manually chaining a visual model, a coding model, and a tool-calling orchestrator.
That can reduce integration complexity, but it does not remove it. Someone still has to supply a browser runtime, a sandbox for code, valid credentials, tool definitions, logging, rate limits, retry behavior, and a policy for dangerous actions.
Why APIs should usually win
For stable, known operations, structured interfaces are generally preferable to screen automation. API calls can be authenticated, versioned, logged, retried, and checked against schemas. A click at pixel coordinates has none of those guarantees.
The integration guidance covered by Unified.to identifies direct APIs, webhooks, and event-driven patterns as standard ways to connect agents with systems. MCP reduces custom integration work by giving models a more consistent way to discover and call tools. [7]
A useful default order is therefore API first, code second, UI last. An API can retrieve an invoice or create a support ticket. Code can reconcile rows, calculate totals, or validate a file. A UI can fill the gaps.
That ordering is not ideological. UI control remains necessary for proprietary desktop tools, old internal systems, and consumer websites where no usable interface exists. It is simply the least stable layer, vulnerable to layout changes, pop-ups, and authentication interruptions.
H Company’s training approach, as outlined by Sam Witteveen, attempts to teach this selection through environments where a task can be completed in more than one way. Reinforcement learning rewards a route that succeeds effectively, rather than clicking by habit.
That is a credible direction, but benchmark scores should be read carefully. A benchmark captures a particular browser, task set, harness, prompt format, and stopping rule. It does not prove an agent will survive an employer’s brittle internal systems.
Personal agents make the loop visible
Abacus AI’s Personal Agents make the same architecture easier to see because they place the agent inside familiar messaging channels. The company’s product page positions Claw and Hermes as personal agents, alongside a larger Abacus AI Agent and Studio environment. [1]
In AI Revolution’s walkthrough of Abacus AI’s launch material, a user messages an agent through WhatsApp or Telegram, while specialist agents exchange context and hand off work. The appealing part is not the chat window. It is delegated coordination.
The video’s production-incident scenario illustrates the desired loop. An agent receives a report, reproduces a bug, checks logs and code, prepares a root-cause analysis, opens a pull request, and stops before deployment for explicit authorization.
That last stop is more important than the automated diagnosis. A deployment changes a live system, potentially affecting customers and data. A sensible agent can investigate broadly but must have sharply limited authority to modify production.
Abacus AI’s pricing is more prosaic than the demos. Its Basic plan costs $10 per user per month and includes limited access to Abacus AI Agent and Studio, with 20,000 credits monthly. [1]
The $20 per-user monthly Pro plan includes Personal AI Agents, full access to Abacus AI Agent and Studio, and 30,000 credits. A separate ChatLLM subscription costs $7 per user monthly. [1]
Those credits matter because heavy reasoning, long agent runs, DeepAgent workflows, and image or video generation consume them. A personal assistant that checks email periodically is cheap. An agent repeatedly browsing, coding, testing, and researching is not equivalent work. [1]
Messaging is not a security boundary
Putting an agent in WhatsApp or Telegram reduces friction, but it does not simplify enterprise governance. A message may trigger access to email, calendars, customer records, source repositories, ticketing systems, or production infrastructure.
That should change how organizations think about the system. ITPro’s reporting argues that enterprises increasingly need to treat agents as digital co-workers, applying safeguards comparable to those used for employees, including monitoring and zero-trust controls. [2]
In practice, this means identity controls, per-tool permissions, approval workflows, immutable logs, scoped credentials, and regular review of what an agent can access. “It knows my preferences” is a convenience feature and a data-governance commitment.
Platform policy adds another constraint. Meta’s WhatsApp Business API policy, effective January 15, 2026, prohibits general-purpose AI chatbots on its business infrastructure, although business-specific customer-service bots remain permitted. [2]
Telegram has not imposed an equivalent general-purpose restriction, according to the research brief. That divergence matters for deployment planning, particularly when an agent’s architecture assumes it can live wherever an employee already messages colleagues.
The AI Revolution video notes that Abacus AI’s WhatsApp connection resembles a linked-device workflow. That may be convenient for an individual, but a company should establish who owns the account, where conversation data resides, and what happens when staff leave.
Coding agents need verification, not confidence
The same action-selection problem appears in software automation. Nate Herk’s AI Automation channel highlights Playwright CLI, which can operate a real browser to test an application as a user would, rather than accepting a coding model’s assertion that a feature works.
That is a useful distinction. Generating code and validating behavior are separate tasks. A model may produce a plausible booking form, but an automated browser test can reveal that payment confirmation fails, a button is hidden, or users can see another account’s data.
Nate Herk also points to Anthropic’s security tooling and repository-based workflows that move specifications and task plans into files. Externalized plans can help a fresh session resume work, rather than relying on a long and increasingly lossy chat history.
But planning files do not eliminate planning errors. Analysis of real agent trajectories cited by Claude Community attributes 19.8 percent of recorded failures to planning drift and 17.6 percent to hallucinated context, such as invented files or configurations. [3]
Long-running work is a particular risk. AgentMarketCap’s discussion of production systems emphasizes failures caused by crashes, context limits, and unreliable tools over multi-hour runs. A task that works for ten minutes may degrade as state accumulates. [5]
The operational response is mundane but effective: split work into checkable stages, persist state outside the model, verify outputs at every boundary, and make retries idempotent. Idempotency means that repeating a command does not create duplicate charges, messages, or records.
Smaller context is a routing decision too
Nate Herk’s second AI Automation video covers Anthropic engineers’ reported reduction of Claude Code’s system prompt by more than 80 percent, without a measurable loss on the company’s internal coding evaluations.
The lesson is not “short prompts are better.” It is that information should be loaded when it is relevant. A video-editing procedure should not occupy context during a database migration, just as a browser screenshot should not be sent when an API response will do.
Anthropic’s reported results remain internal claims, with no independent third-party validation available in the supplied research. They are useful evidence for revisiting accumulated instructions, not proof that every team should delete its project guidance.
Keep durable context that the model cannot infer: the business objective, audience, legal constraints, architecture decisions, ownership boundaries, and known hazards. Remove duplicated rules, obsolete workarounds, and generic instructions that conflict with the current task.
This is where coding automation becomes less glamorous and more valuable. Good agent systems behave less like unrestricted assistants and more like well-designed production pipelines: narrow inputs, explicit tools, observable intermediate states, tests, and accountable humans at decision points.
A practical design for automation
A defensible workflow starts by classifying every action. Reading a CRM record may be low risk. Drafting a reply may require review. Sending that reply, changing a price, merging code, or deploying software should require explicit authorization.
Next, choose the least fragile interface. Use an API or MCP tool where one exists, run code in a constrained environment for data transformation, and reserve UI automation for unavoidable gaps. Log the chosen path and the evidence returned.
Then give the agent a testable definition of done. “Improve the checkout” is not a specification. “Reduce required fields, preserve tokenized payment handling, pass failed-payment tests, and produce a reviewable pull request” is one.
Finally, measure outcomes beyond task completion. Track intervention rate, failed tool calls, retries, cost per completed workflow, time to recovery, permissions exercised, and incidents caught before external impact. Those numbers reveal whether an agent is automating work or merely moving it around.
Frequently Asked Questions
What is the key capability of AI agents in automation?
The key capability of AI agents is selecting the appropriate action among different options such as GUI interaction, code execution, or API calls. This decision-making determines whether automation is robust and effective rather than just impressive in demos. The agent repeatedly inspects the current state, chooses one action, executes it, observes the result, and repeats.
How do AI agents decide which action to take?
AI agents use a model trained to select the best action based on the task and environment. For example, H Company’s Holo4 model chooses between GUI clicks, code, and API calls by evaluating which method completes the task most effectively. Reinforcement learning rewards routes that succeed rather than habitual clicking, helping agents optimize their action selection.
Why are APIs preferred over UI automation for AI agents?
APIs are preferred because they provide stable, authenticated, versioned, and logged interfaces that can be retried and checked against schemas. UI automation, such as clicking pixels, is less reliable due to layout changes, pop-ups, and authentication interruptions. Therefore, a recommended integration order is API first, code second, and UI automation only as a last resort.
What are the challenges in integrating AI agents with legacy systems?
Legacy systems often have outdated architectures or proprietary software that lack usable APIs, forcing agents to rely on fragile UI automation. This increases integration complexity and vulnerability to failures like planning drift, hallucinated context, and degraded performance on long tasks. Human approval steps are essential to mitigate risks when agents interact with such systems.
How do personal AI agents like Abacus AI work and what are their costs?
Abacus AI’s Personal Agents operate within messaging channels like WhatsApp or Telegram, coordinating tasks by exchanging context and delegating work. They follow the agent loop but stop for human authorization before irreversible actions. Pricing starts at $10 per user per month for limited access (Basic plan), while the $20 Pro plan adds Personal AI Agents and 30,000 monthly credits for resource-intensive tasks.
How we researched this
This article was assembled from 4 video sources across 3 channels, 8 cited references.
Nothing here is based on hands-on testing. Where a figure or finding appears, it belongs to the source cited beside it, and the writing says so rather than implying otherwise. Every source is listed below so you can check it.
Sources
I Tried 100+ Github Repos for Claude. These 6 Are the Best — Nate Herk | AI Automation
Anthropic Engineers Just 10x'd Everyone's Claude Code — Nate Herk | AI Automation
Abacus AI's New Personal Agents Are Creating a New Kind of Internet — AI Revolution
Holo4: A Model That Clicks, Codes and Calls Tools — Sam Witteveen
Abacus Personal Agents | Agents that message you back. | Abacus.AI
Beyond the Leaderboard: The 6 Failure Modes That Kill Real Agents — Claude Community
What Are The Main Problems and Limitations With AI Agents For Business Automation? - Rox
AI agents are going rogue - here's what you need to know if you use ChatGPT, Gemini or Claude
Watch Advances and Challenges in AI Agents and Automation on Youtube
Also from the sources
Related Articles
AI Agents Data Governance and API Integration Best Practices
Explore AI agents data governance and API integration to ensure reliable, real-time responses in specialized domains.
AI for Business Productivity: Enhancing Workflows and ROI
Learn how AI boosts business productivity by optimizing workflows, marketing, and data integration with measurable ROI.
AI Agent Development and Multi-Agent Systems Best Practices
Learn AI agent development essentials, from single-agent workflows to multi-agent systems and tool integration challenges.
OpenAI Rogue AI Agents and Security Challenges Explained
Explore OpenAI rogue AI agents, sandbox vulnerabilities, and security challenges from recent incidents like DseWiki and Hugging Face breaches.