A common situation in automation projects: the app that needs to be operated has no exposed API, or the API that exists doesn't cover the specific task at hand. The instinct is often to reach for screen automation as a blanket solution, but that framing skips over a more useful question, which of several interface layers actually fits this specific task with the least fragility.
An API is built for software-to-software communication: defined actions, structured data, predictable responses when something goes wrong. Without one, an automated system has to work with whatever a human user would see and interact with instead, a web page's underlying structure, an accessibility layer that exposes control names and states, a desktop application's window elements, or in the least structured case, screen pixels interpreted through image recognition.
That shift changes the nature of the engineering problem rather than making it simpler. Instead of sending a request and receiving a defined response, a system working through an interface has to observe the current state, choose one action, perform it, then verify the outcome actually matches what was expected before moving on. A button can move. A page can load slowly. A session can expire mid-task. None of that shows up as a clean error the way a failed API call would, it shows up as the interface simply not matching what the system expected, and the system needs to notice that gap rather than proceeding on a false assumption.
Several distinct methods exist for this kind of interface-level interaction, and they are not interchangeable. Browser automation, working directly with a page's underlying structure and load state rather than raw coordinates, tends to be the most reliable option for stable web applications, though it can still break on dynamic content, embedded frames, or expiring sessions. Accessibility-tree interaction relies on the same semantic information assistive technology uses, control names, roles, and current values, which works well when an application implements accessibility properly and considerably less well when it does not. Screen and image-based interaction is the most broadly applicable method, since it does not depend on any particular underlying structure, and correspondingly the least predictable, useful as a fallback for legacy or highly custom interfaces rather than a default choice.
The practical guidance that holds up across all of these methods: use the most structured approach the task actually permits, rather than defaulting to the most broadly capable one. Broad capability and reliability tend to move in opposite directions here.
A separate and more important question sits underneath all of this: whether a task is authorized in the first place. The absence of a public API does not override a platform's terms of service or an account owner's actual permissions, and that question needs to be settled before any interface method is chosen, not treated as an afterthought once a technical path is found. Two boundaries deserve to be treated as hard limits rather than obstacles to route around. Authentication challenges such as CAPTCHAs or multi-factor prompts should pause an automated process and return control to a person, never be treated as something to defeat. And actions with real consequences, sending a message, submitting a form, changing an account setting, moving funds, should require a human confirmation step before executing, regardless of how confident an automated system's internal reasoning appears to be.
Systems built this way also need a different kind of instrumentation than API-only systems typically require: captured screen state at each step, a record of what was observed and what action was taken, the ability to replay a session when something goes wrong, and explicit verification after each action rather than an assumption that it succeeded. The useful evaluation question for any system claiming to handle apps without an API is not whether it can technically interact with a control. It is what happens when that control is not where the system expected, and whether a person reviewing the outcome can actually see what was attempted and why.
More on this approach: https://aidenai.io
