From Trace to Insight: Harness Engineering in SmartPerfetto AI Agent
This article documents the key technical decisions made while building SmartPerfetto from scratch – why certain approaches were chosen over others, where pitfalls were encountered, and how the design was adjusted afterward.
Version note (reviewed October 2026): This remains a retrospective of March–April 2026. Code and counts use the April 10 f3206864 snapshot, not the latest interface. It contains 165 .skill.yaml files including the loader-skipped _base.skill.yaml template, hence 164 analysis definitions. It exposes up to 20 MCP tools in full mode, and 3 in lightweight mode. Original diagrams, session excerpts, latency, and token figures are historical material without a complete trace/configuration/evaluation set; they are not fresh test results. The project is now open source; see the v1.16.1 update at the end.
Why Build This Tool
I have spent years working on Android performance optimization. A significant portion of my daily work involves analyzing Perfetto traces – Perfetto is Google’s open-source system-level tracing tool that captures frame rendering, thread scheduling, CPU frequency, Binder IPC, and other data, making it essentially the standard tool for Android performance analysis. Its trace_processor parses traces and exposes SQL queries, using SQLite in execution without storing all trace data as ordinary SQLite tables.
The trace analysis process is highly repetitive: locate the problem interval, inspect frame data, examine thread states, trace the blocking chain, and correlate system metrics. The steps are similar each time, but the details differ for every trace. This pattern of “fixed workflow, varying details” is well-suited for an AI Agent – automate the data collection and initial attribution within the fixed workflow, while humans make the final judgments and confirmations.
SmartPerfetto is the product of this experiment. It adds an AI analysis panel on top of the Perfetto UI, where users ask questions in natural language (e.g., “analyze scrolling performance”), and a Claude Agent uses MCP (Model Context Protocol, Anthropic’s tool-calling protocol) behind the scenes to execute SQL queries via trace_processor, autonomously completing multiple rounds of data collection and analysis.
The purpose of this article is to document the engineering decisions and lessons learned during the construction process. From the initial “just call the API directly” approach to the current system with up to 20 MCP tools (9 always-on + 11 conditionally injected) + 164 YAML Skills + a three-layer verification system, every design choice was driven by concrete counterexamples – approaches that were tried and failed before being replaced. These hard-won lessons can be directly applied by engineers building AI Agent applications or Android performance tools.
Opening: Two Analysis Paths for the Same Trace
A scrolling trace from a 120Hz device, where the user reports occasional stuttering during list scrolling. Opening it in Perfetto reveals 18 dropped frames during the inertial scrolling phase, with 3 Full-severity frames (~60ms, where the per-frame budget on a 120Hz device is 8.33ms).
Jank and frame duration: FrameTimeline compares expected/actual timelines and presentation outcomes. An actual slice is not CPU rendering time; exceeding one refresh period alone does not prove visible stutter. At 120Hz the period is about 8.33ms, while app deadlines, pipeline latency, target frame rate, and refresh mode need separate checks.
jank_typeis a framework classification. Buffer Stuffing indicates extra latency associated with buffered frames, not necessarily a full queue or proof that the app is blameless.
Path A: Manual Analysis
1 | 1. Open Perfetto UI, drag the timeline to locate the scrolling interval |
thread_state records a thread’s scheduling state (Running / Runnable / Sleeping / Uninterruptible Sleep, etc.). Different states point to different investigation directions – Runnable typically indicates CPU scheduling issues, while Sleeping typically indicates waiting/blocking issues. The
waker_utidfield records the source thread ID that woke up the thread, helping trace cross-process blocking chains.
Steps 3-4 represent the bulk of the work – 18 dropped frames, each requiring expanding thread_state, tracing the blocking cause, and correlating CPU scheduling. The analysis process is serial, frame by frame: the drill-down path may differ for each frame (Binder? Lock? GC? IO?), and all must be examined before drawing conclusions.
Path B: SmartPerfetto Agent
The following condensed excerpt from the original session_agent-1774679540422 illustrates orchestration after “analyze scrolling performance.” Parameters and outputs are pseudocode, not copyable MCP requests. The recorded 18-frame count must not be combined with the seven-row screenshot below as one fully reproduced run.
1 | classifyScene("analyze scrolling performance") -> scrolling (<1ms, keyword matching) |
Metrics snapshot (from logs/metrics/): 16 tool calls, 0 failures, SQL average 652ms.
The following diagram shows the request lifecycle of a complete analysis – every step from user input to final conclusion:
This is a historical design illustration: read submit_analysis_plan as the snapshot’s submit_plan, and use resolve_hypothesis to update hypothesis status. Checkmarks, timings, and model confidence are not acceptance evidence. Sub-agent, tool, and model availability depends on configuration.
Both paths follow the same analysis steps – query frame data -> locate jank -> trace blocking chains -> correlate system state -> synthesize conclusions.
The difference is: manual analysis processes frames serially, with each frame requiring manual expansion and tracing; the Agent uses the scrolling_analysis Skill to batch-retrieve structured data for all 18 frames through a composite Skill, then selects representative frames for deep-drill blocking chain analysis.
The Agent’s analysis results are also rendered directly in the Perfetto UI:
- Auto-Pin: Key frames and slices mentioned by the Agent are automatically marked on the timeline
- Click-to-Navigate: Timestamps and frame IDs in conclusions support click-to-navigate to the corresponding position in Perfetto
- Data Tables: Complete performance data for all 18 frames is rendered as sortable, filterable structured tables
Screenshot: SmartPerfetto frontend exists as a Perfetto plugin
The old UI shows a Deepseek label and connection messages. It illustrates plugin placement, not successful completion by Claude agentv3.
Screenshot: During scrolling analysis, each dropped frame is analyzed in detail – click the arrow on the left to expand
The table shows seven rows, including one UNKNOWN responsibility value. It does not establish analysis or attribution of all 18 frames in the excerpt.
Screenshot: Scrolling analysis conclusion, representative frame analysis
This is historical model output. 95.9% Running supports predominantly running time in the window, not pure computation or a proof that maximum frequency cannot help. Those require profiling, critical-path evidence, and comparative measurements. Frequency changes alone do not establish governor or thermal causes.
Screenshot: Scrolling analysis conclusion, representative frame analysis
cache_miss … makePipeline suggests pipeline/cache work, not proof that all of it is shader compilation. Nested slice durations must not be added directly. “The next six frames” describes this sample, not a guarantee of no later recurrence.
Screenshot: Each analysis round has its own analysis report; content completeness requires a separate check
This March 30 report shows both zero findings and two hypotheses. It proves neither absence of problems nor verified hypotheses. Report/UI data parity needs version-specific checks; see the later v1.16.1 fixes below.
Analysis conclusions, data tables, and the Perfetto timeline are all on the same interface. After the Agent completes batch data collection and initial attribution, engineers confirm key findings directly in the Perfetto UI.
It should be noted that the current Agent still requires human judgment in complex edge cases (misdiagnosis issues are discussed in detail later). This article documents the engineering decision-making process behind building this Agent.
Part 1: Why Can’t an LLM Directly Analyze a Trace?
Before discussing the architecture, a fundamental question must be answered first: why can’t we simply send trace data to an LLM and let it analyze? The answer to this question determines the starting point of SmartPerfetto’s entire architecture.
Data Scale: Traces Don’t Fit in Context
Trace size and event count depend on device, duration, and recording configuration. The original 50–500MB, millions of events, and several GB of expanded text describe sample scales, not a fixed conversion. Binary file size does not directly determine model tokens.
The original “200K tokens ≈ 150KB” was incorrect: tokens-to-bytes varies with language, text, and tokenizer. Context limits also depend on model, version, and access mode. Larger contexts still need room for tool definitions, conversation history, and output.
SmartPerfetto therefore keeps traces in Trace Processor and queries the relevant process, time window, and fields on demand. This controls context cost and preserves a path back to the source data; it does not imply that an LLM can never read a small trace summary directly.
Precise Computation: LLMs Are Poor at Numerical Processing
The daily work of performance analysis revolves around precise numerical values: P50/P90/P99 percentiles of frame durations, VSync period estimation (checked against local refresh modes and valid samples), CPU utilization percentages, and time proportions of various thread states.
LLMs frequently make errors with these numerical computations. A real example: during early testing, Claude judged a 16.7ms frame duration as “normal, not exceeding VSync period” – it was using the 60Hz (16.67ms) frame budget. But this trace was captured from a 120Hz device, where the per-frame budget should be 8.33ms, making 16.7ms actually twice over budget. Such errors may seem small, but in performance analysis they lead to completely opposite conclusions.
SQL and fixed statistical logic make arithmetic reproducible and reduce model arithmetic errors. Correctness still depends on query semantics, data completeness, and function versions. Percentile APIs may use different 0–1 or 0–100 scales, and a single period for the whole trace cannot represent refresh-mode changes.
Domain Knowledge: LLMs Know It but Can’t Apply It
The complexity of Android’s rendering pipeline exceeds many developers’ expectations. The three most common rendering paths are: the standard HWUI pipeline (HWUI is Android’s default hardware-accelerated rendering engine, where View drawing instructions are generated on the main thread, submitted to the GPU by RenderThread, and finally composited to the screen by SurfaceFlinger), historical Flutter UI/raster paths (host RenderThread involvement depends on embedding), and WebView’s multithreaded Chromium rendering path. Beyond these, there are also Jetpack Compose, game engines, camera pipelines, and more. SmartPerfetto’s architecture detection system currently identifies 24+ rendering pipelines, and jank analysis for different pipelines requires examining different threads and metrics – which is why architecture detection is the first step of analysis.
The root cause of stuttering may span threads (main thread blocked -> cause is in RenderThread), span processes (App waiting -> system_server’s WindowManagerService responds slowly), or even span hardware layers (CPU scheduled to a little core -> insufficient compute power -> frame timeout).
The LLM’s training data includes these concepts – it “knows” what RenderThread is, what Binder is, what SurfaceFlinger is. But when facing a specific trace, it lacks the ability to apply this knowledge in a phased, scene-specific manner. For example, when analyzing scrolling jank, one needs to first check frame-level data (which frames dropped, what type of jank), then choose different deep-drill paths based on the predominant root cause type (App-side blocking goes to blocking_chain_analysis, compositor-side delays go to SurfaceFlinger analysis). This kind of step-by-step, conditional-branching analysis workflow requires strategy injection for guidance.
These are observations and routing choices from that period, not universal limits of models. 1.ui/1.raster are historical Flutter thread names, not constants across engine versions; newer engines include platform/UI thread merging. WebView rendering also spans renderer, compositor, raster/GPU threads and processes, not only CrRendererMain. Routing needs version, surface/layer, and event evidence; missing signatures do not establish absence.
Reliability: Error Rates Are High in Practice
Even with the data access problem solved, having an LLM directly produce performance analysis conclusions still faces reliability issues. In SmartPerfetto’s actual operation, I observed several typical output problems:
- Hallucinations: Generating data or metrics that don’t exist in the trace
- Omissions: Missing critical checks (e.g., not checking the impact of JIT compilation and class loading when analyzing startup performance)
- Shallow attribution: Stopping at “main thread busy” without further tracing whether it’s busy with futex (lock contention), binder_wait (cross-process waiting), or GC pause
- Inconsistent conclusions: Analyzing the same trace twice and getting different severity assessments
The three-layer verification section discusses the historical ~30% observation and its sample limitations; it is not a generalizable error rate.
SmartPerfetto’s Division of Labor
Based on these four problems, SmartPerfetto’s architecture divides responsibilities as follows:
1 | LLM (Claude) is responsible for: Tool system is responsible for: |
The LLM handles reasoning and expression; the tools handle querying and computation. Connecting the two is MCP (Model Context Protocol, Anthropic’s tool-calling protocol) – Claude calls trace_processor to execute SQL, invokes YAML Skills for structured analysis, and queries Perfetto stdlib modules through standard MCP interfaces. Analysis results are streamed to the Perfetto UI frontend in real-time via SSE (Server-Sent Events).
The engineering infrastructure supporting this division of labor includes: scene routing (injecting different analysis strategies based on the user’s question), data compression (controlling the volume of data returned to the LLM), and quality verification (intercepting the LLM’s domain misjudgments). The following sections discuss each of these in detail.
The diagram below shows the complete system architecture, illustrating the 4 stages from user request to analysis conclusion:
The diagram’s 17 tools omit the comparison tools; the snapshot’s full mode has 9 always-on plus up to 11 conditional tools. Ports, models, cache-hit rates, and timings in the old diagram are configuration/example values, not current defaults or performance guarantees.
Part 2: From Workflow to Agent
The Difference Between Workflow and Agent
In their December 2024 publication Building Effective Agents (authors Erik Schluntz, Barry Zhang), Anthropic classified AI systems into two categories:
- Workflow: LLM and tools are orchestrated through predefined code paths. What each step does and where the next step goes are all defined by the developer in advance.
- Agent: The LLM dynamically drives its own process and tool usage, autonomously deciding how to complete the task.
The practical significance of this distinction lies in the trade-off between flexibility and controllability. Workflows provide predictability, suitable for tasks with fixed steps; Agents provide flexibility, suitable for open-ended problems that require adjusting direction based on intermediate data. Andrew Ng’s description is apt: rather than making a binary judgment about whether a system is an Agent, think of it as varying degrees of Agent-ness. SmartPerfetto’s agentv2 and agentv3 correspond to opposite ends of this spectrum.
Why Performance Analysis Needs an Agent, Not a Pipeline
Performance analysis is not a fixed “input-in, output-out” process – it’s an exploratory reasoning process. Take a real scrolling analysis as an example:
1 | 1. Start with the overview -> find 47 jank frames, P90 = 23.5ms |
Each step’s decision depends on the previous step’s results – all steps cannot be determined before the analysis begins. Pipelines can also branch, loop, and use state machines. Choosing an Agent delegates some open-ended path selection to a model; it does not mean a program cannot implement those decisions.
SmartPerfetto’s design is a hybrid of determinism and flexibility: known scenes (scrolling, startup, ANR, etc.) use Strategy files to constrain mandatory checks, listing required checks that still need verification; but the specific queries and drill-down directions within each phase are autonomously decided by Claude. Unmatched scenes are explored within the available tools, budget, and permissions.
agentv2: A Typical Workflow
agentv2 used DeepSeek as the backend, employing a Governance Pipeline architecture – orchestrated through planner / executor / synthesizer three-phase pipeline, essentially a predefined multi-step workflow (historical commit 367f61ee5: “Replace the 13-step agentv2 governance pipeline with Claude-as-orchestrator”).
This architecture worked well for scrolling analysis of standard Android applications, but encountered problems with non-standard scenarios. For example, some Flutter traces lack usable frame_timeline results because of recording, platform-version, or surface-association limitations, and the pipeline would get empty results but continue executing subsequent steps, ultimately outputting conclusions based on empty data.
agentv3: Migration to Agent Architecture
On March 2, 2026 (commit 367f61ee5), I switched to the Claude Agent SDK. After receiving tool definitions and strategies, Claude autonomously decides which tools to call, in what order, and what data to query.
An AI Agent typically possesses the following characteristics, with agentv3’s implementation mapped as follows:
| Characteristic | Implementation in SmartPerfetto | Code Location |
|---|---|---|
| Autonomy | Agent autonomously decides which tools to call and in what order | claudeRuntime.ts |
| Reasoning | REASONING_NUDGE may be appended to the first 4 relevant data-tool calls | claudeMcpServer.ts |
| Tool Use | Up to 20 MCP tool calls to trace_processor | 9 always-on + 11 conditional |
| Planning | submit_plan + requirePlan() gating | Disabled in lightweight mode |
| Reflection | 3-layer Verifier + Correction Prompt (max 2 rounds) | claudeVerifier.ts |
| Error Recovery | SQL error-correction learning + cross-session misdiagnosis pattern learning | Cross-file |
| Memory | Short-term: Analysis Notes / Artifact Store; Long-term: Pattern Memory / SQL Fix Pairs | Several stores, not a fixed layer count |
1 | agentv2 (Workflow): Fixed pipeline -> missing empty-result branch -> possible wrong conclusion |
9 Rounds of Review After Migration
From March 2 to March 20, 9 rounds of architecture review were conducted. The most impactful rounds:
| Round | Date | Key Findings |
|---|---|---|
| Round 1 | 3/2 | 12 fixes after initial SDK integration – SQL knowledge base not wired into System Prompt, CPU core count hardcoded to 4 in jank_frame_detail |
| Round 3 | 3/12 | Architecture wiring audit – 12 instances of “implemented but not connected,” e.g., verification pipeline skipped when 0 findings |
| Round 7 | 3/15 | Perfetto Stdlib integration – preloaded modules 4->22, Schema Index 708->761 |
| Round 9 | 3/20 | Production quality audit after 18 days of real traces – 3 P0 + 4 P1 + 5 P2, which catalyzed the three-layer verification system |
These rounds and issue counts come from my development notes at the time. Commit contents establish changes, not that every review or real trace passed. Old commit identifiers have been replaced with resolvable identifiers matched by commit contents.
Cold Start 4-Layer Cascading Bug
On March 19, 2026 (commit b184547a1), cold starts were found to be incorrectly classified as warm starts. Tracing revealed this was a cascading issue across 4 layers:
1 | Layer A (Perfetto Stdlib): bindApplication's ts is ~98ms earlier than launchingActivity -> excluded by filter |
Fix scope: rewrote 10 downstream Skills, added 4 new startup analysis Skills. This issue demonstrates that in a Skill dependency chain, a semantic error in a single upstream field amplifies through each layer.
Ghost MCP Query – Async Lifecycle Mismatch
Another cross-layer bug caught on April 7, 2026 (commit e8aa4dc65): after an analysis timed out, the session had already been cleaned up and the SSE stream closed, but trace_processor’s stderr continued to emit errors like no such table: cpu_frequency_counters / no such column: ts 90 seconds later – orphan logs with no owner attributable to any session. The two SQL pitfalls in Error 4 below were traced back from this pile of stderr output.
The root cause was in the SDK Query’s async lifecycle:
1 | Layer A (Claude Agent SDK): SDK's internal AsyncIterator still producing messages |
Early exit from for await normally invokes the iterator’s return() (see AsyncIteratorClose); a generator’s finally can clean up resources. “Break only affects the consumer” is therefore not a language-wide rule. Here, the wrapper failed to reliably propagate closure to the SDK Query. The fix exposed a { stream, close } object so timeouts and finally explicitly close the underlying query. Closing must be idempotent, and work already dispatched to other processes still depends on their cancellation contracts. The following lifecycle sketch omits configuration and error handling:
1 | // claudeRuntime.ts |
This bug was particularly insidious because the surface symptom (orphan SQL error logs) and the true root cause (async resource lifecycle mismatch) were far apart – the error messages looked like “Agent wrote bad SQL,” but the reality was “the Agent had long since stopped; it was the SDK subprocess that hadn’t.” In Agent applications, “breaking out of a loop” is often not as clean as one might imagine; any long-lived async resource needs an explicit close channel rather than relying on for-await‘s automatic cleanup.
Part 3: Three Key Engineering Decisions
Decision 1: Scene Classification – From Full Injection to On-Demand Loading
Initially, I stuffed the analysis strategies for all 12 scenes (scrolling / startup / ANR / interaction / pipeline / game / memory, etc.) into the System Prompt, totaling 15000+ tokens. The logic was: Claude should know all scene analysis methods so it can handle whatever the user asks.
After running in practice, Claude was found to confuse terminology across scenes – when analyzing scrolling, it referenced startup-phase metrics, mixing up VSync intervals (inter-frame timing) with bindApplication (process initialization). One possible contributor was overlapping terminology across scenes: “frame” in the scrolling context means a rendering frame, while in the startup context it means the first frame display. When all 12 strategy sets are present simultaneously, choosing the wrong context becomes easier.
The solution was to implement scene classification, injecting only one strategy set at a time:
1 | // sceneClassifier.ts -- 12 scenes, <1ms execution |
Keywords and priorities are declared in each .strategy.md‘s YAML frontmatter, not hardcoded in the source:
1 | # scrolling.strategy.md |
A .strategy.md file is the declaration entry point. After development reloads, confirm server-side registration and any supporting capabilities.
The historical record reports a drop from roughly 15000 to 4500 estimated prompt tokens, with fewer observed scene mix-ups. These project estimates are neither billing measurements nor a hard cap on every conversation. A declarative scene can start with .strategy.md; additional Skills, output support, and verification depend on the feature.
When multi-turn conversations accumulate substantial context (analysis notes, historical plans, pattern memory, etc.), the System Prompt may again exceed the 4500-token budget. At that point, lower-value paragraphs are progressively dropped by priority: SQL knowledge base references -> trace-completeness guidance -> historical analysis experience -> historical lessons learned -> SQL error-correction pairs -> sub-agent collaboration guidelines -> historical analysis plans. Core paragraphs (role, methodology, scene strategy, output format) are never dropped.
If trimming optional sections still exceeds the budget, the snapshot logs a warning and keeps core sections. Thus 4500 is a soft budget. Dropping completeness guidance must not erase unknown/missing evidence in the query layer. Development reloading depends on server caching/file-watching; refreshing a browser alone does not reload every configuration.
Decision 2: Artifact Store – Controlling Data Volume Returned to the LLM
Decision 1 solved the System Prompt bloat problem. But even with only one strategy set injected, each Skill call during Agent execution still produces large amounts of data (200+ rows of frame data), and putting all of it into context creates new problems.
Early versions returned complete Skill execution results (e.g., 200 rows of frame data, 487 rows of blocking analysis) to Claude. Each Skill result was approximately 3000 tokens, and with 5-8 Skill calls per analysis, Skill data alone consumed 15000-24000 tokens.
Token cost was one concern, but a more surprising discovery was: the more data Claude received, the worse its output quality became. When faced with 200 rows of frame data, it tended toward line-by-line description (“Frame 1 took 12.3ms, Frame 2 took 15.7ms…”) rather than pattern-level summarization. I suspect the reason is that when the context is flooded with large amounts of numbers, the LLM’s attention becomes scattered.
The solution was to store Skill results in the ArtifactStore, returning only compact references (~440 tokens) to Claude – row count, column names, and summary information. When details are needed, Claude retrieves them on demand via fetch_artifact with pagination. Structured results are sent to the frontend through SSE (Server-Sent Events) data events, bypassing the LLM entirely.
1 | invoke_skill("scrolling_analysis") execution result: |
Three granularity levels for fetch_artifact:
| Level | Returns | Approx. Tokens |
|---|---|---|
summary |
Row count + column names + first row sample | ~50 |
rows |
Paginated data (offset/limit) | ~200-500 |
full |
Original structure, capped at 500 rows in this snapshot; marked truncated above that | Content-dependent |
After the adjustment, each Skill’s token cost dropped from ~3000 to ~440, and 8 Skills went from ~24000 to ~3520 tokens. Claude’s output shifted from line-by-line description to pattern-level summarization, while the frontend still received complete data for table rendering.
“Full” means data admitted by the execution-result contract, not the entire trace. full itself caps rows at 500; inspect truncated/totalRows and paginate beyond that. Reading a summary is not reviewing all data. The ~440/3000-token figures are sample estimates; content, pagination, and caching change total consumption. Data can use separate events within the same analysis SSE stream, not necessarily a separate connection.
Decision 3: Three-Layer Verification – Lessons from Real Misjudgments
About 18 days after agentv3 went live, I reviewed quality on March 20, 2026; the corresponding fix is 1c64fd8fb. I was surprised by the figure I recorded at the time: “about 30% of conclusions contained errors.” I did not include the sample size, sampling method, labeling criteria, or per-case results. This remains a historical observation, not a model error-rate benchmark or a verifier detection rate.
The examples that prompted rules also require separating heuristics from established facts:
| Original output/rule | What must be checked |
|---|---|
| VSync alignment offset marked CRITICAL | Check refresh mode, time source, and presentation impact; ±0.5ms is not a universal normal threshold |
| Count every Buffer Stuffing frame as App jank, or exclude them all | Separate framework labels, presentation gaps, and App deadlines; buffered-frame latency does not clear the app of responsibility |
| One anomalous frame cannot be CRITICAL | One frame can cause a noticeable freeze; consider absolute delay, interaction, and impact, not recurrence alone |
| Sleeping at 35% (469ms) must be raised from MEDIUM to HIGH | Establish the window, wait cause, and TTID/TTFD critical path; the percentage alone cannot determine severity |
These cases involve rules, measurement semantics, and reasoning, not only missing domain knowledge in training data. The historical verifier used three broad categories:
1 | 1. Heuristics: match known problem patterns |
In claudeVerifier.ts at f3206864, model review can be skipped by heuristic conditions, time out, or produce an unparseable result. It does not independently query the entire trace. Another model checking text is not independent factual verification. Historical passed primarily means the collected issues contain no error; skipped/unknown is not complete acceptance.
Severe issues can trigger up to two correction attempts, with the second conditional on factors such as newly detected errors. Budget exhaustion, timeout, missing evidence, and completed correction need distinct outcomes; generating a conclusion alone is not a pass.
Cross-session patterns are recorded in logs/learned_misdiagnosis_patterns.json. The snapshot loads unexpired patterns with occurrences >= 2, but repetition does not prove a rule correct. For example, the original “TTID 1912ms exceeds dur_ms 1338ms by 43%, so raise severity” first requires matching definitions and start points. Otherwise it learns a mismatch of measurement scopes as a diagnostic rule.
Part 4: Why Not Use a Standard Skill System?
The Design Choice from SOP to YAML Skills
Performance analysis teams typically have their own SOPs (Standard Operating Procedures): how to investigate scrolling jank, how to analyze slow startups, how to diagnose ANR. SOPs are usually documents or checklists – experienced engineers follow them, new team members learn from them.
Claude Code Skills provide instructions in SKILL.md and can include scripts, templates, and reference material. They are not merely parameterized prompts. This comparison concerns the original text-only SOP approach versus the project’s YAML query engine. A standard Skill can also invoke deterministic scripts or SQL.
I initially went down this path. The Prompt given to Claude was: “Query the frame_timeline table, find jank frames, analyze the main thread’s state distribution during jank frames.”
Claude understood the intent without issue, but the SQL it generated was different each time. Sometimes the JOIN path was correct (slice -> thread_track -> thread), other times it wrote slice.utid directly – a column that doesn’t exist. The result format was also inconsistent, sometimes 3 columns, sometimes 5, making frontend rendering impossible.
The reason is simple: SOPs are written for humans. An engineer seeing “query frame_timeline” knows exactly what SQL to write. The LLM’s understanding of Perfetto’s SQL schema is incomplete (these schemas have limited coverage in training data), and each translation from SOP text to SQL introduces variance.
SmartPerfetto’s YAML Skills took a different approach – not Prompt templates, but declarative SQL execution units:
1 | # YAML Skill: SQL predefined, result format fixed |
This teaching fragment omits Skill input declarations. thread_state has a direct utid, not track_id. It clips states to a half-open window and excludes unknown/unfinished dur <= 0 rows. Percentages use observed state time within that window; missing time is not zero. Resolve and validate the UTID and nanosecond bounds before substitution instead of interpolating arbitrary text.
The comparison is between freely generating SQL on each run and invoking predefined queries. Generated workflows can also be evaluated, constrained, and composed; predefined queries make it easier to fix datasets, parameters, and output schemas for deterministic regression tests. YAML itself does not guarantee correctness: time windows, units, identity, and missing data still need review.
| Dimension | Prompt Template (SOP-style) | YAML Skill (Declarative Execution) |
|---|---|---|
| SQL source | LLM generates dynamically each time | Predefined in YAML, parameter substitution |
| Result format | May differ each time | Fixed column names and types |
| Regression testable | Can be evaluated; generation variance needs control | Fixed queries; six traces describe the historical sample set |
| Frontend rendering | Requires parsing free text | Schema-driven tables/charts |
| Composable | Agent/script orchestration is possible | Composite skills call atomic skills |
| Vendor adaptation | Requires different Prompts | .override.yaml overrides SQL |
The final division of labor: Claude is responsible for understanding intent, selecting Skills, and reasoning about attribution; YAML Skills are responsible for precise SQL queries and structured output. Claude invokes Skills via invoke_skill, Skills return structured data, and Claude makes judgments based on the data.
Why Not Expose Each Skill as an Independent MCP Tool?
A natural question: why not directly register all 87 atomic analysis capabilities as 87 MCP Tools and let Claude call them directly?
Trying this in practice reveals a problem: the MCP tool list grows linearly with the number of tools. Eagerly including all 87 names, schemas, and instructions consumes context. Tool-selection difficulty was also observed, but without a controlled evaluation this does not establish a universal relationship between tool count and accuracy.
SmartPerfetto’s design gives Claude only 2 Skill-related MCP Tools:
invoke_skill(skillId, params)– Execute a specified Skilllist_skills(category?)– Query available Skills by scene category
Capabilities are discovered on demand via list_skills(category="scrolling"), then invoked with invoke_skill. 2 MCP Tools encapsulate 160+ analysis capabilities, and the number of top-level tool schemas stays fixed.
Another benefit is that the YAML format lowers the contribution barrier. If a performance analysis expert has experience with a particular analysis scenario, they can directly write a YAML Skill defining SQL queries and output formats, without needing to know TypeScript or modify backend code. Development loading can shorten iteration; still confirm that the server loaded the definition and validate its output.
This overhead describes eagerly loading every schema, not an MCP requirement to resend all tools each turn. Tool search, deferred loading, and prompt caching change the cost, and more tools do not inherently reduce accuracy. Two entry points keep top-level schemas stable, but discovery results and outputs still consume tokens. YAML-only additions work when existing steps, output types, and actions suffice; new capabilities can still require engine changes and validation.
Skill System Structure
The Skill count grew from a dozen in the early project to 164 at that point. The growth was not driven by “as many as possible,” but by continuously encountering new scenarios that needed coverage in analysis practice – initially there was only frame analysis for standard HWUI, then Flutter applications required specialized Skills, then vendor differences required overrides, then startup analysis needed independent detection logic for JIT, class loading, and Binder pool respectively.
Analysis definitions by directory at f3206864, excluding _base.skill.yaml (definition count is not independent scenario count):
| Type | Count | Location | Description |
|---|---|---|---|
| Atomic | 87 | skills/atomic/ |
Single detection capabilities (VSync period, CPU topology, GPU frequency, GC events, etc.) |
| Composite | 29 | skills/composite/ |
Multi-step combined analyses (e.g., scrolling_analysis orchestrates multiple atomic Skills) |
| Pipeline | 28 | skills/pipelines/ |
Rendering pipeline detection + teaching (24+ Android rendering architecture identification) |
| Module | 18 | skills/modules/ |
Module-categorized analyses (app / framework / hardware / kernel) |
| Deep | 2 | skills/deep/ |
Deep analysis (CPU profiling, callstack analysis) |
Additionally, skills/vendors/ contains .override.yaml for 8 vendors (Pixel / Samsung / Xiaomi / Honor / OPPO / Vivo / Qualcomm / MTK), overriding vendor-specific SQL in generic Skills.
Layered Results
Early Skill output was flat – a Skill returned one large table, with 200 rows of frame data mixed together. Users would see all the data at once upon opening, with no sense of hierarchy. In practice, engineers’ reading habits follow a pattern: first look at the summary (what’s the jank rate, what’s P90), then decide whether to expand the details, then deep-drill into specific frames.
Now Skill output is organized by layer, with progressive frontend rendering:
1 | summary -- "47 jank frames, P90=23.5ms, SEVERE accounts for 12%" |
Each Skill step declares its display layer via display.level (supported values and counts vary by snapshot and are not a stable interface). The frontend automatically renders tables and charts based on column types (timestamp, duration, percentage, bytes, etc.) and interaction actions (navigate_timeline to jump to the trace position, navigate_range to select a time range, copy to copy data) from the DataEnvelope – a Skill reusing supported types/actions usually needs no additional frontend code. This is key to how 164 Skills coexist while frontend code volume remains manageable.
Step Types
Initially, all Skills had only one step type: execute a single SQL query. Later, scenarios requiring composition of multiple Skills arose (e.g., scrolling_analysis needs to first query frame data, then perform blocking analysis on each jank frame), as well as scenarios requiring iteration over data rows (per-frame diagnosis), leading to gradual expansion of step types:
| Step Type | Description | Usage Frequency |
|---|---|---|
atomic |
Single SQL query, the most basic step type | Most common |
skill |
References another Skill’s result, for reusing existing capabilities in composite analysis | 56 occurrences |
iterator |
Iterates over data rows, executing sub-queries for each row | Used in 5 composite Skills |
diagnostic |
Diagnostic step, generating structured diagnostic conclusions | 38 occurrences |
parallel |
Execute multiple steps in parallel (code supports it, not yet used in Skills) | 0 |
conditional |
Branch selection based on conditions (code supports it, not yet used in Skills) | 0 |
iterator is the core of per-frame analysis – for example, for the 8 most severe out of 18 jank frames, execute blocking_chain_analysis on each one individually, analyzing the blocking cause for each frame independently. parallel and conditional are defined in the type system but not yet used by any Skill – this is because current analysis scenarios are already covered by skill references + iterator traversal. When more complex scenarios are introduced (such as multi-path parallel data collection), these will come into play.
Domain Skill Examples
The following examples illustrate why so many specialized Skills are needed – each Skill addresses a specific problem that a generic approach can’t handle.
Consumer Jank Detection – Framework Labels != User Perception
jank_type, presentation cadence, and perceived smoothness are different measurements. An absent framework label does not guarantee a smooth interaction; longer presentation gaps may also reflect a lower target frame rate, static content, or the end of an animation.
The historical consumer_jank_detection aimed to supplement framework labels with a VSync estimate and a 1.5-period threshold, treating gaps above six periods as session breaks. Its implementation has material limits:
vsync_configsnaps to selected refresh rates, while the interval query separately computes an unsnapped period. Missing VSYNC-sf/expected data falls back to 60Hz, which is an assumption, not a device measurement or variable-refresh model.- It uses
PERCENTILE(..., 0.5)as a median. Perfetto’s PERCENTILE scale is 0–100; a median uses 50. The variable namevsync_mediandoes not establish correctness. present_tsis a proxy constructed fromactual_frame_timeline_slice.ts + dur. LAG partitions byupid, with deduplication by process and token. It does not universally establish same-layer physical scanout times. Multiple layers, duplicate tokens, and dropped/unfinished frames require further checks.
This describes the historical heuristic and its limits, not independent perceptual ground truth or a correct jank count for every refresh mode. This article revision neither changes the SmartPerfetto repository nor reruns that algorithm.
Blocking Chain Analysis – Cross-Thread, Cross-Process Root Cause Tracing
The root cause of a single dropped frame may involve a multi-layer causal chain:
1 | Frame 42 took 62ms (budget 8.33ms) |
The blocking_chain_analysis Skill provides key evidence for this chain via 3 SQL steps: main thread state distribution (Running / Sleeping / IO proportions) -> waker tracing (finding who woke the main thread via waker_utid) -> blocking function aggregation (cumulative time for futex / binder_wait / io_schedule). This kind of cross-layer analysis is very difficult to implement stably by having Claude write SQL on its own with generic prompts.
This is a chain to verify. waker_utid identifies an observed waker, not automatically a lock owner or Binder server; a futex wait is not necessarily mutex contention. Establish each causal edge using lock events/stacks, Binder transaction relationships, remote thread states, and timing.
Flutter Architecture Branching – Different Rendering Modes Require Different Analysis Logic
These are two historical Android embedding surface choices, not all Flutter renderer modes. Skia/Impeller and SurfaceView/TextureView are separate dimensions. Thread names in this table belong to older samples:
| Mode | Jank Analysis Target Threads | Goes Through Host RenderThread |
|---|---|---|
| TextureView (dual pipeline) | 1.ui + 1.raster + RenderThread | Yes |
| SurfaceView (single pipeline) | 1.ui + 1.raster | No |
If standard HWUI analysis logic is used to analyze a Flutter SurfaceView application, 1.raster thread’s duration would be incorrectly attributed to RenderThread. SmartPerfetto automatically identifies Flutter applications through architecture detection (24+ rendering pipelines) and switches to the dedicated flutter_scrolling_analysis Skill.
But “automatically identifying Flutter” itself also hit a snag (commit efb464acf, 4/6). The early pipeline detector scored each architecture independently, with the highest score winning – as a result, Flutter TextureView traces were frequently misclassified as STANDARD. The reason was that Flutter TextureView’s host side still goes through the HWUI pipeline (Choreographer#doFrame / DrawFrame / RenderThread), and these signals were absorbed by both the STANDARD and TEXTUREVIEW classifiers. STANDARD had broader signal coverage (traces almost always contain Choreographer frames), so its total score often exceeded the specialized TEXTUREVIEW, misclassifying Flutter apps as STANDARD. The same problem occurred with WeChat Skyline (absorbed by WEBVIEW) and game engines (absorbed by STANDARD/MIXED).
The fix added exclusions to relevant generic candidate YAML: TEXTUREVIEW_STANDARD exits when Flutter/WebView signals appear; STANDARD_LEGACY/MIXED/SURFACEVIEW_BLAST exclude game signals; OPENGL_ES excludes WebView/game, and WebView candidates exclude Flutter. A TEXTUREVIEW object does not actively disable all STANDARD candidates. 24+ pipelines cannot score independently – they need a “specialized -> generic” priority chain. This is a classic example of “once you have many pipelines, you must implement mutual exclusion” – and it’s why, after the Skill count grew to 160+, simply “routing to the correct Skill” itself became an independent engineering problem.
For newer engines, check the Flutter architecture documentation and actual configuration. Platform/UI thread merging on Android/iOS invalidates a requirement for a separate 1.ui; Platform Views composition also changes the path. A host RenderThread does not prove every Flutter frame uses it, and SurfaceView does not make the whole page a single pipeline.
Vendor Overrides – Same Metric Has Different Field Names on Different Platforms
In traces from Qualcomm, MediaTek, and Google Tensor, the same metric has different field names (e.g., GPU frequency is called gpufreq on Qualcomm but might be gpu_freq_khz on MediaTek). .override.yaml lets the same Skill automatically adapt its SQL across different platforms, without needing to write independent Skills for each vendor.
gpufreq/gpu_freq_khz are example track/event names to inspect, not universal Qualcomm/MediaTek column contracts. Units, sampling semantics, device, and kernel version also matter. All eight snapshot overrides are startup.override.yaml; their existence does not prove universal vendor GPU-frequency support.
Part 5: SQL Engineering
The Skill system discussed above ultimately comes down to SQL queries – each Skill step executes predefined SQL. SQL is SmartPerfetto’s core – all performance data retrieval ultimately goes through SQL queries to trace_processor. This section expands on several engineering aspects at the SQL level: query pattern design, official stdlib reuse, Schema indexing, result compression, and error-correction learning.
SQL Query Patterns: Time Interval JOINs and Recursive Bucketing
Slices, sched, and thread_state commonly represent intervals with a start and duration. Relating frames to possible blocking calls requires their intersection; filtering only starts inside a frame misses waits crossing its boundary.
This teaching query assumes jank_frames and blocking_calls are caller-defined relations already restricted to the relevant process/thread and analysis range, not built-in Perfetto tables. Overlap is a correlation clue, not proof that the call blocked the frame.
1 | WITH overlaps AS ( |
Filter the raw nanosecond overlap above 0.5ms before rounding to two display decimals. The original non-aggregate HAVING was invalid here and is replaced with an outer WHERE. Nanosecond representation does not imply nanosecond measurement accuracy. Candidate intervals may nest; adding their overlaps can double-count blocked time.
The original bucketing example also used undefined bucket_ns and overlap_dur values and could truncate the window at bucket 30. This complete observed-running-time example uses Perfetto v58.3 fields sched.utid/ucpu/ts/dur. Rounding the positive integer width upward covers the whole window in at most 30 buckets:
1 | WITH RECURSIVE |
${...} denotes validated integers substituted by the caller, not native Perfetto SQL parameter syntax. Rows report observed Running time per bucket and ucpu. Buckets without sched rows are omitted, not assumed idle or fully captured. Big/little grouping additionally requires a verified capacity/topology mapping with an unknown category. The original _cpu_topology is created by the project’s cpu_topology_view Skill, not a stable public stdlib contract from which prime/big/medium/little can be assumed across devices.
The Skill Engine has its own ${param|default} template syntax. Engine defaults do not replace type/range validation or SQL-injection boundaries.
Perfetto Stdlib Reuse
Perfetto officially maintains a SQL standard library (stdlib) that provides numerous predefined views and functions. For example, the android_frames view encapsulates multi-table join logic for frame rendering data, and the _android_critical_blocking_calls internal table aggregates critical blocking calls. Using these official abstractions directly is far more stable than hand-writing SQL from base tables.
SmartPerfetto’s stdlib integration went through several iterations – one rollback even completely reversed the “direction of optimization”:
Initial phase: Only 4 stdlib modules were preloaded (android.frames.timeline, android.binder, android.startup.startups, android.input), with most Skills’ SQL querying base tables directly. The advantage was fast startup; the downside was Skills everywhere hand-writing multi-table JOINs.
Round 7 (3/15): Expanded the preload set to 22 modules, including
linux.cpu.utilization,android.garbage_collection,android.oom_adjuster,slices.with_context, covering CPU/GC/OOM/slice and other commonly used dimensions. The logic at the time was: batchINCLUDEall commonly used stdlib modules at trace load time, no repeated module initialization for subsequent queries, which still have execution cost.Rollback to lazy loading (4/1, commit
9d313df5a): Large-trace loading had producedsocket hang up. The fix reduced eager preload to three Tier-0 modules, loaded lazily and serially before queries, with bounded retries. This supports reducing concurrent initialization pressure; the disconnect alone does not prove SQLite schema mutation caused it. Trace Processor is not simply an ordinary SQLite database:1
2
3
4
5
6
7// workingTraceProcessor.ts
// Tier 0: absolute minimum stdlib modules needed for any analysis to start.
const CRITICAL_STDLIB_MODULES = [
'android.frames.timeline', // 19 skills reference -- frame/jank foundation
'android.startup.startups', // 16 skills reference -- startup foundation
'android.binder', // 22 skills reference -- IPC/blocking foundation
];These 3 were selected by “skill reference count” as the highest-frequency dependencies. Other stdlib modules are now declared on-demand via Skill YAML’s
prerequisitessection or explicitINCLUDE PERFETTO MODULEin SQL when first used.On-demand discovery:
perfettoStdlibScanner.tsscans the Perfetto source directory to automatically discover all available modules, exposed through thelist_stdlib_modulesMCP tool for Claude toINCLUDEnon-preloaded modules on demand.
1 | // perfettoStdlibScanner.ts -- scans perfetto/src/trace_processor/perfetto_sql/stdlib/ |
This change addressed that loading bottleneck. Lazy loading reduces initialization work but moves some latency to first use. Stable, frequently used modules can still benefit from prewarming; choose based on connection behavior, trace size, and measurements.
A separate lesson: using the stdlib’s android_garbage_collection_events view is far more stable than manually JOINing slice + thread + process tables to query GC events – because GC event slice names vary across Android versions (concurrent mark sweep vs young concurrent copying vs HeapTaskDaemon), and the stdlib has already handled these compatibility issues. But stdlib views have their own pitfalls (column name prefixes, modules not auto-loaded), which are expanded upon in the SQL error-correction section below.
Reusing stdlib reduces duplicate logic without guaranteeing cross-version stability. Underscore-prefixed internal objects such as _android_critical_blocking_calls are not stable public APIs. Modules discovered from a source checkout must match the Trace Processor binary actually running.
SQL Schema Index: Letting Claude Know What Tables Are Available
Perfetto trace_processor contains hundreds of tables and views, plus stdlib modules – Claude cannot possibly remember them all. The lookup_sql_schema MCP tool provides a search interface for Claude to look up relevant tables, views, and function definitions by keyword.
Under the hood is an index file automatically generated from the Perfetto source code (perfettoSqlIndex.light.json), containing 761 templates, each recording the name, category, type (table/view/function), column definitions, and parameters.
Queries use tokenized matching + score-based ranking:
- Name/category/description containing the full search term -> high score
- Multi-word queries match by token separately -> >= 50% token match required to count as relevant
- Underscore-separated table name segments support prefix matching (“frame_time” matches “frame_timeline_slice”)
- Returns top 30 results
Combined with sqlKnowledgeBase.ts‘s intent mapping, bilingual queries are also supported: user input of “jank” maps to search terms like ['jank', 'frame', 'dropped'], and input of “startup” maps to ['android_startups', 'launch', 'time_to_display']. When multiple intents match simultaneously, scores are additive – for example, querying “startup frame jank” triggers both the startup and jank intents simultaneously, giving the highest scores to templates matching their intersection.
SQL Result Compression
When Claude queries directly via execute_sql, it can pass a summary=true parameter to trigger result compression. The compression logic is implemented in sqlSummarizer.ts:
Numeric columns: Compute min, max, avg, and percentiles (P50 / P90 / P95 / P99), letting Claude understand the data distribution without seeing raw rows.
String columns: Count the top 5 values and their occurrence frequencies, providing the data’s category distribution.
Sample row selection: Select 10 representative sample rows from the complete result. The selection strategy is: if the data contains performance-related columns like dur, latency, jank, count, sort descending by that column and take the top 10 (the most severe data is usually most analytically valuable); if there’s no clear performance metric column, use evenly-spaced sampling.
1 | -- 200 raw result rows (~3000 tokens) compressed to: |
This works in conjunction with the Artifact Store mentioned earlier – the Artifact Store compresses Skill results (data returned by invoke_skill), while the SQL Summarizer compresses results when Claude executes SQL directly. Together, two layers of compression cover both paths through which the Agent obtains data.
Statistics cover rows supplied to the summarizer, not automatically the whole trace. Interpret them with query limits, filtering, nulls, and truncation. The worst ten rows are extreme samples, not an unbiased population sample; percentile interpolation and numeric precision must match the SQL definition. The JSON illustration omits fields and contains comments, so it is not directly parseable.
SQL Error-Correction Learning
Claude is not fully familiar with Perfetto’s SQL schema and writes incorrect queries. The following are typical errors actually recorded (from logs/sql_learning/error_fix_pairs.json):
Error 1: JOINed on a non-existent column
Perfetto’s slice table has no direct utid column. To associate slice with thread, the intermediate table thread_track is needed: slice.track_id -> thread_track.id -> thread_track.utid -> thread.utid.
1 | -- Error: no such column: s.utid |
Error 2: Ambiguous column names
1 | -- Error: ambiguous column name: name (both slice and process have a name column) |
Error 3: Treating counter samples as intervals
counter stores timestamps and values, not dur. AVG(value) is a sample mean; with uneven sampling it is not a time-weighted mean and is not an equivalent fix.
For a time-average frequency, partition by track, derive intervals with LEAD(ts), include the last valid sample before the window, clip boundaries, and compute SUM(value * overlap_ns) / SUM(overlap_ns). Extending the final sample to the window end requires an explicit sample-and-hold assumption. Uncovered time remains unknown. CPU-frequency intervals are also available from linux.cpu.frequency below; check the kHz unit.
Error 4: stdlib table name / column name pitfalls
Commit a262dadf1 added these two pitfalls to the methodology prompt template. It did not establish that runtime learning generated two error/fix pairs. The clue was the orphan stderr after a timeout: the analysis had already ended, but trace_processor was still spitting out errors. Following those logs led back to these SQL problems.
1 | -- Error 4a: no such table: cpu_frequency_counters |
Filters are omitted above; first INCLUDE PERFETTO MODULE android.garbage_collection; for the GC query. gc_dur describes a GC interval, not a main-thread stop-the-world pause. Associate running, runnable, and pause intervals separately. Fields are checked against Perfetto v58.3 frequency and garbage_collection.
Stdlib view authors often use domain-prefixed column names to avoid ambiguous column issues during multi-table JOINs, but Claude defaults to assuming conventional names like ts/dur. This “stdlib’s own naming conventions vs. general SQL conventions” conflict can’t be fully solved automatically through schema introspection – the lookup_sql_schema tool can tell Claude the table exists and what columns it has, but it can’t predict “this time Claude will definitely assume ts.” Actual-schema context, column constraints, and result-semantic checks also help; error/fix pairs are supplementary.
The detection and learning mechanism for these errors works as follows: when SQL execution fails, the error message and SQL are temporarily stored; when a subsequent SQL execution succeeds, the system uses Jaccard similarity matching (excluding SQL structural keywords like SELECT/FROM/WHERE, and Perfetto common tokens like utid/dur/slice) to determine whether it’s a corrected version of the same query. The matching threshold is >30%, with a 60-second time window. A successful match generates an error->fix pair that is persisted to disk.
When a new analysis begins, up to 10 error-correction pairs are loaded, with up to 5 rendered into the System Prompt, so Claude can see these known pitfalls before writing SQL. Error-correction pairs have a 30-day TTL and are automatically cleaned up upon expiration – Perfetto’s SQL schema changes with version updates.
Successful execution means SQL was accepted, not that the fix preserved intent or produced correct values. Similarity/time-window matching can pair unrelated queries, and a 30-day TTL does not bind a pair to a schema version. Recheck learned fixes against the target trace and engine.
Part 6: The Evolution of the Development Process Itself
This final section steps slightly outside the product itself to discuss the development process. SmartPerfetto was built with AI-assisted development – from the first line of code to now, Claude Code has been the primary programming tool. Looking back over these three months, the way I use AI-assisted development has itself gone through several iterations, with logic similar to SmartPerfetto’s evolution from agentv2 to agentv3.
Stages of AI-Assisted Development
First, a brief explanation of the tools and concepts involved:
- Claude Code: Anthropic’s CLI tool that enables terminal conversations with Claude, where Claude can directly read/write files and execute commands. Throughout development, I ran with
--dangerously-skip-permissions(dangerous mode) and bypass permissions enabled, allowing Claude to autonomously execute file edits, command runs, Git operations, etc. without per-action confirmation. This dramatically improved iteration speed – Claude can continuously execute complete “modify code -> run tests -> check results -> fix -> re-run” cycles without being interrupted by permission dialogs, at the cost of requiring the developer to have sufficient trust in Claude’s operations and perform after-the-fact review. - Claude Agent SDK: Anthropic’s Agent development framework, on which SmartPerfetto’s agentv3 backend is built. The SDK encapsulates multi-turn conversation management, MCP tool-calling loops, automatic context compression (auto-compact), and other capabilities. Developers define the tool set and System Prompt, and the SDK drives Claude to autonomously complete multi-round analysis.
- Plan Mode: Claude Code’s planning mode, where the AI first outputs a structured implementation plan (which files to modify, what to change, order and dependencies), the human reviews and confirms, then the AI executes the code changes.
- Superpowers: A third-party development workflow/Skill collection, not the name of Claude Code’s entire plugin ecosystem or of Chrome DevTools/Playwright MCP. The personal narrative retains the original “SuperPower” wording. Plan Mode is a Claude Code feature.
- Codex + Codex MCP: Codex is OpenAI’s coding-agent product/tool family, not one fixed model name. The historical workflow used a local MCP integration to request review from another Agent.
codex mcp-serveris a CLI integration entry point; read-only behavior depends on tools and sandbox configuration, not MCP itself. See the CLI reference. - Agent Team / Subagent: Subagents can have separate contexts and roles. Current Agent teams and subagents are distinct mechanisms; launching parallel subagents is not automatically the full team feature.
- Skills / Hooks: Claude Code’s extension mechanisms. Skills are reusable task templates (e.g.,
/commit,/simplify), and Hooks are scripts that automatically execute on specific events (e.g., before/after tool calls).
My Actual Evolution
Stage 1: Direct Conversation
The earliest development approach was describing requirements directly in Claude Code and letting the AI modify code. Similar to pair programming where one person talks and the other writes. At this stage, every change needed line-by-line human review because the AI’s understanding of project context was limited, frequently making local modifications that didn’t align with the overall architecture.
Stage 2: Plan Mode (SuperPower)
After adopting Plan Mode, the workflow became: I describe the requirement -> AI outputs a structured implementation plan (which files to modify, what to change in each, modification order and dependencies) -> I review the plan -> AI executes after confirmation. This shifted the review focus from “examining code line by line” to “reviewing the architecture plan,” noticeably improving efficiency.
Stage 3: Plan Mode (SuperPower) + Peer Review (Codex)
Relying on a single AI to generate plans is prone to blind spots. I began sending the plan to Codex for independent review after the Plan Mode plan was finalized. Codex accesses the codebase in read-only mode and provides feedback from three angles: architectural soundness, edge cases, and missed risks. This effectively introduces a code review step into the AI development workflow.
Most of the 9 rounds of architecture review mentioned earlier in the article went through this process. Taking the Perfetto Stdlib integration as an example (Round 7, March 15), Codex reviewed 3 rounds, raising a cumulative 36 feedback points, including issues I had missed in the plan such as the stdlib module preloading strategy and the Schema Index cache invalidation mechanism.
Stage 4: Harness-ified Engineering Pipeline
By the later stages, the development workflow had become:
1 | 1. I determine requirements and architectural direction |
In this workflow, human involvement is concentrated at Step 1 (requirements and architectural decisions) and Step 4 (evaluating review feedback). Code details, test execution, and formatting cleanup are handled by the engineering pipeline.
This is the historical SmartPerfetto workflow, not a universal command list for every checkout. The article does not supply a complete terminal record of these runs. A script existing, an SSE stream ending, or a report being written does not establish correct attribution. Reproduction needs a pinned commit, trace, model/configuration, expected evidence, and terminal outcome, distinguishing deterministic checks, real-model E2E, failures, and skips.
Correspondence with SmartPerfetto’s Architecture
Looking back, my AI-assisted development workflow and SmartPerfetto’s Agent analysis workflow are structurally similar:
| Dimension | SmartPerfetto Agent Analysis | My AI-Assisted Development |
|---|---|---|
| Intent Understanding | Scene Classifier identifies the scene | I determine the requirement direction |
| Strategy Injection | .strategy.md injects analysis methodology | Plan Mode outputs implementation plan |
| Execution | MCP tool calls for SQL/Skills | Claude Code executes code changes |
| Quality Verification | 3-layer Verifier (heuristic + Plan + Haiku) | Regression tests + Codex review |
| Correction Loop | Correction Prompt has Claude revise | Test failure -> analyze -> fix -> re-run |
| Cross-session Learning | Pattern Memory + SQL error-correction | CLAUDE.md rule accumulation + memory system |
Both systems evolved in the same direction: human involvement shifted from the execution layer up to the decision layer. SmartPerfetto went from a fixed pipeline (humans define every step) to autonomous reasoning (humans define goals and constraints); my development approach went from line-by-line review to reviewing architecture plans.
This is not coincidental – the core of Harness Engineering is building sufficient engineering infrastructure (tests, verification, review) so that humans can trust the AI’s execution results and focus their attention on higher-level decisions.
The diagram below summarizes SmartPerfetto’s complete Harness Engineering landscape – from input routing to cross-session learning:
The original diagram is retained for structure. “~30% false positives caught” is not supported by a detection-rate evaluation and cannot be inferred from the original observed ~30% error figure. An 80% cache-hit rate, 85% savings, and improving accuracy over time are not verified guarantees. The snapshot uses 60/90-day TTLs for positive/negative patterns respectively, not one universal 90-day TTL.
Conclusion
Looking back over three months of iteration, from agentv2’s 13-step fixed pipeline to agentv3’s autonomous reasoning, from observed misjudgments to three-layer verification, from a 15000-token System Prompt to 4500-token on-demand loading – every change was driven by a concrete failure experience.
After completing this project, I have two takeaways about AI Agent application development.
The first is: the bulk of the work is not in the LLM API calls themselves, but in the engineering infrastructure surrounding the LLM:
- How should the System Prompt be organized so the LLM doesn’t confuse context? -> Scene classification + on-demand loading + token budget
- How do you control the LLM’s execution order so it thinks before acting? -> Planning Gate + hypothesis submission
- How much data should be returned to the LLM? -> Artifact Store: give summaries, not full data
- How do you detect and intercept the LLM’s domain misjudgments? -> Three-layer verification + correction loop
- How do you ensure data query precision? -> YAML Skills (declarative SQL) + SQL error-correction learning
- How do you adapt to different rendering architectures and chip platforms? -> Architecture detection + vendor overrides
The second takeaway is: the Agent’s “environment” matters far more than prompt wording. In agentv3’s early days, I spent considerable time adjusting System Prompt wording and formatting. I later discovered that what truly affects Agent output quality is not how the prompt is written, but what tools it’s given, what data is returned, and what constraints are imposed. Three concrete examples:
- After adding
submit_plangating, Claude stopped making directionless SQL queries (previously it would make purposeless queries likeSELECT * FROM slice->SELECT * FROM thread), and analysis paths became organized - After adding ArtifactStore, the data Claude received dropped from 200 rows to summary references, and reasoning focus noticeably improved
- After adding the
lookup_knowledgetool, root cause analysis depth progressed from “main thread blocked” to “Binder counterpart system_server had delayed response due to CPU being scheduled to a little core”
None of these improvements were achieved through adjusting prompt text – they were achieved by changing the Agent’s tool set and data environment. If I had one piece of advice for engineers building AI Agent applications, it would be to put more effort into tool design and data control, and rely less on prompt wording to solve these problems.
Future Directions
This is the April 2026 roadmap; the original “closed-source, not released” status is obsolete. At that point I was developing it individually, with the following directions:
- Deep vendor integration – The current 8 vendors’
.override.yamlonly covers core Skills. More vendor-specific metrics (Qualcomm Snapdragon Profiler data, MediaTek MAGT signals, Samsung GameOptimizing service) need to be integrated one by one. - CI integration + batch processing – Moving from interactive analysis to automatically analyzing performance regressions for each build in CI pipelines. Including unattended mode, result comparison against baselines, and automatic regression tagging.
- E2E verification framework – The current 6-trace regression test verifies the correctness of Skill data output, but does not verify the quality of Agent conclusions. An E2E verification system is needed: given a trace + known root cause -> check whether the Agent correctly identified it.
- Codebase integration – Mapping slices/functions in the trace back to source code locations, combined with git blame to locate the change that introduced the issue.
October 2026 update (v1.16.1): The project is now open source, with a Web UI, CLI, reports, and portable runtimes. Use the v1.16.1 README and release record for that version rather than the retrospective’s old tool list, model, or startup settings.
The release explicitly fixes retention of HTML-report rows, timelines, nested content, and comparison/batch data, plus offline interactions. Generating a report and preserving all evidence in it are separate acceptance criteria. Source, documentation, and release records are inspectable; this article review has not rerun every device, model, or trace scenario.
Historical source entry points: MCP registration, Artifact Store, and System Prompt. These pinned links explain the April design; newer behavior requires its corresponding tag.
SmartPerfetto Discussion Group
The original discussion-group QR code is retained as a historical image; its validity and available places are not verified. Current bug reports can use the project issue tracker.
For contact details, see About the author.