AI desktop agent
Automiq
A Windows agent that turns plain-language instructions into visible, verifiable desktop actions.
University final-year project · 2026
Open the report and prepare a concise reply.
- Screen perceived
- Action planned
- Verifying result
Privacy-safe interface composition based on the shipped product.
My contribution
Built the working product end to end: Electron interface, FastAPI sidecar, vision-agent loop, environment discovery, auth, history, templates, and execution safety.
The challenge
What the system had to solve
General desktop automation has to understand changing interfaces, act at the right moment, and prove that each step worked without becoming a rigid macro.
Product surface
Modules and functionality
- 01
Typed and voice command input
- 02
Screenshot-driven planning loop
- 03
Mouse, keyboard, app, and browser execution
- 04
Environment and browser-profile discovery
- 05
Run HUD, history, templates, and context
- 06
OTP authentication and local persistence
Engineering
Decisions that shaped the build
- Structured the agent as a perceive → plan → act → settle → repeat loop.
- Added content protection and click-through behavior so the agent cannot operate its own window.
- Built browser-profile routing, duplicate-launch guards, scroll-loop protection, and UI-settle detection.
- Supported English and Roman Urdu voice commands through server-side Gemini transcription.
Outcome
What the work delivered
- Produced a functional Windows desktop product with a packaged portable build.
- Made live automation understandable through streamed steps, logs, and a persistent HUD.
- Turned an academic concept into a demonstrable agent with production-minded safety controls.
Technology
Core stack
- Electron
- React
- Python
- FastAPI
- SQLite
- Gemini