Android Perfetto Series 17: Domain Automation, Issue Checklists, and Platform...

Word count: 7.3kReading time: 45 min

 2026/05/04

For slow camera opens, audio underruns, a stalled WebView first screen, or dropped frames in Flutter and games, the evidence is scattered across the app, system_server, cameraserver/audioserver, HAL, SurfaceFlinger, and scheduling tracks. Simply inspecting a few more tracks in the UI can easily leave critical timestamps unnoticed.

Using Camera and Audio as examples, this article describes a reusable method for domain-specific analysis: capture the right data, identify the objects, calculate stages and blocking, and preserve timestamps in the report so that reviewers can return to the UI. Camera and Audio are examples; the core is a domain schema and collaboration with platform tracing. By the end, you should at least be able to break a slow camera open into reviewable stages and an audio underrun into cycle anomalies, rather than delivering only a table of the longest slices.

Perfetto Series Catalog

This article applies the preceding methods to domain automation: the SQL methods from Part 11, quality checks from Part 12, application semantics from Part 13, and field capture from Part 15 all come together here in specific domains and platform infrastructure. Rather than explaining those methods again, we reorganize them around domain objects.

Domain analysis preserves objects and review points

Illustration: Camera stages belong to the same operation; Audio periods are measured from actual cycles. Reports retain timestamps for returning to the UI.

Domain Analysis Starts with an Object Dictionary

The difficulty with Camera and Audio is not whether a camera/audio category exists, but which objects are actually present in the trace:

Domain	Objects	Why identify them?
Camera	App client, `cameraserver`, provider, HAL, request/session IDs	Open, configure, preview, and capture can span different processes
Audio	App audio thread, `audioserver`, MixerThread, FastMixer, audio HAL, Bluetooth/audio route	Underruns often result from cycle jitter, irregular writes, HAL blocking, or scheduling delays

The first step in automated analysis is to generate an object dictionary: relevant processes, threads, tracks, slice names, log tags, and request/session IDs. Subsequent SQL does not guess a root cause directly; it calculates stage durations and blocking evidence around this dictionary.

The object dictionary should be script output, rather than a set of keywords that exists only in someone’s head. A Camera case may involve the app process, cameraserver, provider@2.7-service, a vendor camera daemon, a codec, and SurfaceFlinger. An Audio case may involve the app, audioserver, audio.primary, Bluetooth, and a media codec.

An initially confirmed object dictionary looks like this. SQL is only one way to generate and validate it:

object_dictionary_version: camera_audio_v1
objects:
  - role: camera_app_client
    object_type: thread
    process_name: com.example.app
    thread_name: main
    track_id: 102
    uid: 10123
    package_name: com.example.app
    service_instance: ""
    hal_transport: ""
    selinux_context_or_domain: untrusted_app
    vendor_owner: app-team
    source_signal: track_event
    confidence: 1.0
    owner_confirmed: true
  - role: camera_service
    object_type: process
    process_name: cameraserver
    vendor_owner: platform-camera
    source_signal: atrace
    confidence: 1.0
    owner_confirmed: true

The domain schema is the collaboration interface between the app, platform, SQL, and reports. Version it rather than scattering it across articles, scripts, and verbal agreements:

schema_version: camera_domain_v1
domain: camera
scenario: camera_open_preview
owner_team: platform-camera
privacy_level: system-metadata-and-app-markers
object_roles:
  - camera_app_client
  - camera_service
  - camera_provider
  - camera_hal
  - display_service
stable_events:
  - event_name: Camera#OpenStart
    owner: app
    required_args: [camera_id, session_id, operation_id, event_id]
  - event_name: Camera#ConfigureStreamsStart
    owner: platform
    required_args: [camera_id, session_id, operation_id, stream_id]
  - event_name: Camera#FirstPreviewFrame
    owner: platform
    required_args: [camera_id, session_id, operation_id, stream_id, surface_id]
  - event_name: Camera#OpenEnd
    owner: app
    required_args: [camera_id, session_id, operation_id, event_id, result]
  - event_name: Camera#ConfigureStreamsEnd
    owner: platform
    required_args: [camera_id, session_id, operation_id, stream_id, result]
  - event_name: Camera#FirstHalBuffer
    owner: platform
    required_args: [camera_id, session_id, operation_id, stream_id]
stage_definitions:
  - stage_name: open_api
    start_event: Camera#OpenStart
    end_event: Camera#OpenEnd
  - stage_name: first_preview_marker
    start_event: Camera#OpenStart
    end_event: Camera#FirstPreviewFrame
report_fields:
  - stage_name
  - status
  - start_ts
  - end_ts
  - dur_ms
  - evidence_slice
  - review_ts
  - source_role

The script first lists candidate objects, then the domain owner confirms which belong to this scenario. That confirmation must be recorded in a domain_objects table or equivalently structured YAML/JSON, rather than remaining in someone’s head. Subsequent queries join only against this dictionary.

CREATE TABLE domain_objects AS
SELECT
  'candidate' AS role,
  'thread' AS object_type,
  p.name AS process_name,
  p.upid,
  p.pid,
  NULL AS uid,
  NULL AS package_name,
  th.name AS thread_name,
  th.utid,
  th.tid,
  tt.id AS track_id,
  NULL AS log_tag,
  NULL AS request_id_key,
  NULL AS session_id_key,
  NULL AS service_instance,
  NULL AS hal_transport,
  NULL AS selinux_context_or_domain,
  NULL AS source_signal,
  NULL AS vendor_alias,
  NULL AS vendor_owner,
  0.5 AS confidence,
  0 AS owner_confirmed
FROM thread th
LEFT JOIN process p USING (upid)
LEFT JOIN thread_track tt USING (utid)
WHERE LOWER(p.name) LIKE '%camera%'
   OR LOWER(p.name) LIKE '%audio%'
   OR LOWER(p.name) LIKE '%media%'
   OR p.name IN ('com.example.app', 'cameraserver', 'audioserver',
                 'surfaceflinger', '/system/bin/surfaceflinger');

Note that this uses ordinary SQLite CREATE TABLE, not CREATE PERFETTO TABLE: the latter creates a read-only table, so UPDATE cannot write back the results of manual confirmation. At a minimum, the fields must cover role, process/thread, track_id, vendor aliases, confidence, and manual confirmation status. The initial version can still be generated from keywords, but reports may use only objects with owner_confirmed = 1. Otherwise, as soon as a vendor HAL, provider, codec, or Bluetooth route changes, the automation falls back to fuzzy matching.

List candidate objects for the domain owner to review as follows:

SELECT DISTINCT
  role,
  process_name,
  pid,
  thread_name,
  tid,
  confidence,
  owner_confirmed
FROM domain_objects
ORDER BY process_name, thread_name;

A person needs to inspect this output. Automation can narrow the scope, but LIKE cannot reliably establish ownership for every provider, HAL, codec, and Bluetooth route.

Write confirmed objects back to the same table, or persist them as equivalently structured YAML/JSON. The following confirms objects by upid already reviewed in this trace. Do not automatically confirm everything matching %provider%: that would bring other Camera or Media services into the report. Audio needs its own role confirmation. If only specific threads in a process belong to a role, also restrict by utid.

-- 42/43/44 are placeholders: replace them with upid values actually reviewed above.
WITH confirmed(upid, role) AS (
  VALUES (42, 'camera_app_client'),
         (43, 'camera_service'),
         (44, 'camera_provider')
)
UPDATE domain_objects
SET owner_confirmed = 1,
    role = (SELECT role FROM confirmed c WHERE c.upid = domain_objects.upid)
WHERE upid IN (SELECT upid FROM confirmed);

Subsequent SQL uses only objects with owner_confirmed = 1. The candidate table aims for completeness; the confirmed table makes reports stable. The fixed role enum includes camera_app_client, camera_service, camera_provider, camera_hal, display_service, audio_app, audio_server, audio_mixer, audio_hal, and bluetooth_audio.

Start Configuration with a Domain Preset

First make the permission boundaries explicit:

Capability	Who can use it	Notes
App markers / metadata	Ordinary apps	`android.os.Trace`, NDK ATrace, Track Event, application metadata
Registered triggers	Ordinary apps or test harnesses	Can activate only trigger names predeclared by the platform
App profiling	shell + `profileable` or debuggable	Prefer `profileable` for release performance comparisons; use debuggable only for debugging
ftrace/process stats/FrameTimeline/system log	shell, Traceur, trusted platform/OEM consumers	Subject to SELinux and build-type restrictions; ordinary apps cannot enable these
Uploads, rate limits, privacy, preset selection	Platform/OEM diagnostics workflow	Requires authorization, redaction, retention periods, and auditing

The following is a lab_domain_debug configuration, not a low-overhead field preset. It retains sched/freq/idle, Camera/Audio/hal/binder atrace, FrameTimeline, process stats, and logs for development devices, userdebug, or laboratory reproduction. Reduce the field version to necessary data sources; enable logs, binder_lock, and high-frequency domain events only for short, focused traces.

Non-Pixel devices on Android 9/10 may first need traced enabled. Before Android 12, without root, do not assume perfetto can read a configuration directly from /data/local/tmp; prefer stdin. Measuring the first camera frame visible to the user also requires SurfaceFlinger/FrameTimeline. Camera/HAL signals alone measure only API return, a HAL buffer, or a provider stage.

duration_ms: 30000
flush_period_ms: 10000

buffers {
  size_kb: 131072
  fill_policy: RING_BUFFER
}

data_sources {
  config {
    name: "linux.ftrace"
    ftrace_config {
      compact_sched { enabled: true }
      ftrace_events: "sched/sched_switch"
      ftrace_events: "sched/sched_waking"
      ftrace_events: "power/cpu_frequency"
      ftrace_events: "power/cpu_idle"
      atrace_categories: "camera"
      atrace_categories: "audio"
      atrace_categories: "gfx"
      atrace_categories: "view"
      atrace_categories: "hal"
      atrace_categories: "binder_driver"
      atrace_categories: "binder_lock"
      atrace_categories: "am"
      atrace_categories: "wm"
      atrace_categories: "input"
      # Use categories/events supported by the target device Record page or --query output.
      atrace_apps: "<target_package>"
    }
  }
}

data_sources {
  config {
    name: "android.surfaceflinger.frametimeline"
  }
}

data_sources {
  config {
    name: "linux.process_stats"
    process_stats_config {
      scan_all_processes_on_start: true
      proc_stats_poll_ms: 5000
    }
  }
}

data_sources {
  config {
    name: "track_event"
    track_event_config {
      disabled_categories: "*"
      enabled_categories: "camera"
      enabled_categories: "audio"
    }
  }
  # Replace with an App/platform producer name actually registered on the device; repeat as needed.
  producer_name_filter: "<confirmed_producer_name>"
}

data_sources {
  config {
    name: "android.log"
    android_log_config {
      log_ids: LID_DEFAULT
      log_ids: LID_SYSTEM
      log_ids: LID_EVENTS
      log_ids: LID_CRASH
      filter_tags: "CameraService"
      filter_tags: "CameraProvider"
      filter_tags: "AudioFlinger"
      filter_tags: "AudioTrack"
      filter_tags: "AudioPolicyManager"
    }
  }
}

atrace_categories enables Android system categories; it does not automatically enable android.os.Trace or NDK ATrace in the app. App-side ATrace shares ATRACE_TAG_APP and requires package-specific atrace_apps configuration, with no separate category filtering. When arguments and cross-thread correlation are needed, use the Perfetto SDK Track Event approach from Part 13.

For app profiling or certain local debugging signals on a user build, release packages should use <profileable android:shell="true" />. debuggable changes performance characteristics and is suitable for debugging, not rigorous timing comparisons. System services, HAL, and CPU/native profiling require userdebug/eng or platform diagnostics permissions through their respective paths.

Official documentation lists android.log support for userdebug. A rooted user build may also capture logs if logd/SELinux permissions allow it, but adb root alone does not guarantee this as a general capability. On ordinary user builds or in production, do not assume logs will be in the trace. Include external logcat, bugreports, or platform-redacted logs in the same case package. Field packages should retain only allowlisted tags and window summaries; raw logs require authorization and redaction.

Category names also depend on the target system. Perfetto documentation separates ATrace into system categories and per-app events. System categories come from Android internal processes; app events share ATRACE_TAG_APP and must be enabled by package name. Do not simply copy the category list from someone else’s device into a preset. Check supported categories for the current version in the Perfetto UI Record page or the device-side configuration.

buffers.size_kb configures the Perfetto central buffer, not the ftrace per-CPU kernel ring buffer. For sched/ftrace data loss, inspect ftrace overrun/dropped/data_loss entries in stats. Adjust parameters such as ftrace_config.buffer_size_kb and drain_period_ms only when necessary, first verifying device compatibility and overhead with a short trace. Before adding vendor kernel tracepoints, check /sys/kernel/tracing/events/*/*/format, field stability, and Perfetto parser/stdlib support. Otherwise, they can only serve as candidate evidence in raw ftrace, or require changes to the Perfetto parser or a custom data source.

Do not express the quality gate as only an abstract score. At a minimum, always run this SQL:

SELECT name, idx, severity, source, value
FROM stats
WHERE value != 0
  AND (
    severity IN ('error', 'data_loss')
    OR name IN (
      'ftrace_setup_errors',
      'ftrace_cpu_has_data_loss',
      'traced_buf_trace_writer_packet_loss',
      'traced_buf_chunks_overwritten',
      'traced_buf_chunks_discarded',
      'traced_buf_patches_failed',
      'traced_flushes_failed',
      'traced_final_flush_failed',
      'track_event_parser_errors',
      'track_event_tokenizer_errors',
      'track_event_thread_invalid_end',
      'graphics_frame_event_parser_errors',
      'systrace_parse_failure',
      'clock_sync_failure',
      'invalid_clock_snapshots'
    )
    OR name GLOB 'ftrace_cpu_*overrun*'
    OR name GLOB 'ftrace_cpu_*dropped*'
    OR name GLOB 'traced_buf_*packet_loss'
  )
ORDER BY name, idx;

The meaning of idx varies by statistic: for traced_buf_* it is usually a buffer index, and for ftrace CPU statistics it is a CPU number; check definitions for other entries. Group findings by affected_signal: ftrace/sched, atrace/systrace, track_event, FrameTimeline, android.log, clock, and central_buffer. Explain overwritten data separately under RING_BUFFER: it may simply mean old data outside the window was overwritten, or that insufficient pre-issue evidence remains. Apply separate report downgrades for data loss, packet loss, and overwrites.

Analyze Camera by Stage

A slow camera open needs more than a statement that “the camera is slow.” Break it into at least these stages:

Stage	Owner	What to examine
App initiates open	App marker	User action, permissions, whether the UI thread is waiting
App to `cameraserver`	App + camera service	Binder round trips, server thread-pool congestion
`cameraserver` to provider/HAL	Platform event	Provider process, HAL threads, device waits
Configure streams	Platform/HAL event	Stream count, resolution, Surface, HAL configure duration
First HAL buffer	HAL/provider event	Request submission, buffer returned from HAL
First presented frame	FrameTimeline/SurfaceFlinger + app marker	Buffer enters SurfaceFlinger and becomes visible to the user
Capture/result	Platform/HAL event	Request ID, result callback, buffer/metadata return

These are three different measurement endpoints: API return, the first frame returned from HAL, and the first frame visible to the user. The first two primarily concern the Camera framework/HAL; the third also involves BufferQueue, SurfaceFlinger, and FrameTimeline. State which one you measured in the report. Otherwise, “open got faster” might mean only that the API returned earlier, with no improvement to the first preview frame.

Domain SQL must be tied to the issue window. That window can come from trigger metadata, application markers, user reproduction steps, or an interval selected manually in the UI. If owner events are missing, report missing_owner_event or stage_unavailable; do not promote a LIKE query into a stage conclusion.

WITH target_window AS (
  SELECT 120e9 AS start_ts, 126e9 AS end_ts
)
SELECT start_ts / 1e9 AS start_s, end_ts / 1e9 AS end_s
FROM target_window;

When stable events exist, first restrict the analysis to a single operation. The following requires all stages to carry the same camera_id/session_id/operation_id, with configure also matching the target stream_id. These IDs are a contract that application instrumentation must provide, not something automatically supplied by the system. No duration is emitted when events are missing, candidates are duplicated, or timestamps are reversed. Use EXISTS to match a slice to objects, avoiding row multiplication from multiple tracks in the object dictionary. Treat FirstPreviewFrame as an application-defined candidate endpoint until layer/token/present has been checked; it cannot yet be called the first user-visible frame:

WITH target_window AS (
  SELECT 120000000000 AS start_ts, 126000000000 AS end_ts
),
target_operation(camera_id, session_id, operation_id, stream_id) AS (
  VALUES ('0', 'session-17', 'open-3', 'preview-1')
),
events AS (
  SELECT s.name, s.ts
  FROM slice s
  LEFT JOIN thread_track tt ON s.track_id = tt.id
  LEFT JOIN thread th USING (utid)
  CROSS JOIN target_window w
  CROSS JOIN target_operation o
  WHERE s.ts >= w.start_ts AND s.ts < w.end_ts
    AND CAST(EXTRACT_ARG(s.arg_set_id, 'debug.camera_id') AS TEXT) = o.camera_id
    AND CAST(EXTRACT_ARG(s.arg_set_id, 'debug.session_id') AS TEXT) = o.session_id
    AND CAST(EXTRACT_ARG(s.arg_set_id, 'debug.operation_id') AS TEXT) = o.operation_id
    AND (
      s.name IN ('Camera#OpenStart', 'Camera#OpenEnd')
      OR CAST(EXTRACT_ARG(s.arg_set_id, 'debug.stream_id') AS TEXT) = o.stream_id
    )
    AND EXISTS (
      SELECT 1 FROM domain_objects d
      WHERE d.owner_confirmed = 1
        AND d.role IN ('camera_app_client', 'camera_service', 'camera_provider',
                       'camera_hal', 'display_service')
        AND (d.track_id = s.track_id OR d.utid = th.utid
             OR (d.object_type = 'process' AND d.upid = th.upid))
    )
),
stage_pairs(stage_name, start_name, end_name) AS (
  VALUES
    ('open_api', 'Camera#OpenStart', 'Camera#OpenEnd'),
    ('configure_streams', 'Camera#ConfigureStreamsStart', 'Camera#ConfigureStreamsEnd'),
    ('first_hal_buffer', 'Camera#OpenStart', 'Camera#FirstHalBuffer'),
    ('first_preview_marker', 'Camera#OpenStart', 'Camera#FirstPreviewFrame')
),
stages AS (
  SELECT
    p.stage_name,
    COUNT(CASE WHEN e.name = p.start_name THEN 1 END) AS start_count,
    COUNT(CASE WHEN e.name = p.end_name THEN 1 END) AS end_count,
    MIN(CASE WHEN e.name = p.start_name THEN e.ts END) AS start_ts,
    MIN(CASE WHEN e.name = p.end_name THEN e.ts END) AS end_ts
  FROM stage_pairs p
  LEFT JOIN events e ON e.name IN (p.start_name, p.end_name)
  GROUP BY p.stage_name
)
SELECT
  stage_name, start_count, end_count, start_ts, end_ts,
  CASE
    WHEN start_count = 0 OR end_count = 0 THEN 'stage_unavailable'
    WHEN start_count != 1 OR end_count != 1 OR end_ts < start_ts THEN 'ambiguous_or_invalid'
    ELSE 'marker_measured'
  END AS status,
  CASE WHEN start_count = 1 AND end_count = 1 AND end_ts >= start_ts
    THEN (end_ts - start_ts) / 1e6 END AS dur_ms
FROM stages;

If stable events have not yet been fully instrumented, fall back to a candidate-slice query. This query only establishes candidate stages. Focus on process_name, thread_name, slice_name, and dur_ms; it cannot directly replace the stage definitions above:

WITH target_window AS (
  SELECT 120e9 AS start_ts, 126e9 AS end_ts
)
SELECT
  s.ts / 1e9 AS ts_s,
  s.dur / 1e6 AS dur_ms,
  s.name AS slice_name,
  th.name AS thread_name,
  p.name AS process_name,
  th.tid,
  p.pid
FROM slice s
JOIN thread_track tt ON s.track_id = tt.id
JOIN thread th USING (utid)
LEFT JOIN process p USING (upid)
CROSS JOIN target_window w
WHERE EXISTS (
    SELECT 1 FROM domain_objects d
    WHERE d.utid = th.utid AND d.owner_confirmed = 1
  )
  AND s.ts < w.end_ts
  AND s.ts + s.dur > w.start_ts
  AND (
    LOWER(s.name) LIKE '%camera%'
    OR EXISTS (SELECT 1 FROM domain_objects d
      WHERE d.utid = th.utid AND d.owner_confirmed = 1
        AND d.role IN ('camera_app_client', 'camera_service', 'camera_provider', 'camera_hal'))
  )
ORDER BY s.dur DESC
LIMIT 100;

If this query does not provide enough stage information, there are usually two options: add android.os.Trace / ATRACE / Track Event instrumentation, or have the platform add stable event names at key stages in CameraService, the provider, and HAL. Validate camera_id/session_id/request_id in event arguments. With Track Event, obtain them from args/debug annotations or flow IDs rather than relying only on event names.

Analyze Audio by Cycle

Audio issues depend even more on periodic behavior than Camera issues. One long slice does not necessarily cause an audible problem. Jitter across several consecutive mixer cycles, irregular app write intervals, and blocking HAL writes are more directly relevant to underruns or dropouts.

An Audio report should first examine whether the tail of the cycle-duration distribution grows, whether anomalies occur consecutively, and whether they coincide with underrun logs, route changes, or the time of an audible problem. The longest slices are supporting clues only.

Audio candidate slices must also be restricted to the issue window and should preferentially use domain_objects, preventing historical noise from the entire trace from entering the analysis:

WITH target_window AS (
  SELECT 120e9 AS start_ts, 126e9 AS end_ts
)
SELECT
  s.ts / 1e9 AS ts_s,
  s.dur / 1e6 AS dur_ms,
  s.name AS slice_name,
  th.name AS thread_name,
  p.name AS process_name
FROM slice s
JOIN thread_track tt ON s.track_id = tt.id
JOIN thread th USING (utid)
LEFT JOIN process p USING (upid)
CROSS JOIN target_window w
WHERE EXISTS (
    SELECT 1 FROM domain_objects d
    WHERE d.utid = th.utid AND d.owner_confirmed = 1
  )
  AND s.ts < w.end_ts
  AND s.ts + s.dur > w.start_ts
  AND (
    LOWER(s.name) LIKE '%audio%'
    OR LOWER(s.name) LIKE '%mixer%'
    OR LOWER(s.name) LIKE '%underrun%'
    OR EXISTS (SELECT 1 FROM domain_objects d
      WHERE d.utid = th.utid AND d.owner_confirmed = 1
        AND d.role IN ('audio_app', 'audio_server', 'audio_mixer', 'audio_hal', 'bluetooth_audio'))
  )
ORDER BY s.ts
LIMIT 200;

Next, examine intervals between Mixer/FastMixer scheduling slices in audioserver. A single mixer work cycle may be preempted into several sched slices, so the start-to-start interval between adjacent Running slices is not an audio mixer period. The following outputs only scheduling intervals and off-CPU gaps. Explaining them still requires correlation with thread_state, HAL slices, underrun logs, and route metadata.

WITH target_window AS (
  SELECT 120e9 AS start_ts, 126e9 AS end_ts
),
mixer_runs AS (
  SELECT
    ss.ts,
    ss.dur,
    th.name AS thread_name,
    p.name AS process_name,
    LEAD(ss.ts) OVER (
      PARTITION BY ss.utid
      ORDER BY ss.ts
    ) AS next_ts
  FROM sched ss
  JOIN thread th USING (utid)
  LEFT JOIN process p USING (upid)
  CROSS JOIN target_window w
  WHERE p.name = 'audioserver'
    AND ss.dur > 0
    AND ss.ts < w.end_ts
    AND ss.ts + ss.dur > w.start_ts
    AND (
      LOWER(th.name) LIKE '%mixer%'
      OR LOWER(th.name) LIKE '%fastmixer%'
    )
)
SELECT
  ts / 1e9 AS ts_s,
  thread_name,
  dur / 1e6 AS running_ms,
  (next_ts - ts) / 1e6 AS sched_start_interval_ms,
  (next_ts - (ts + dur)) / 1e6 AS off_cpu_gap_ms
FROM mixer_runs
WHERE next_ts IS NOT NULL
ORDER BY sched_start_interval_ms DESC
LIMIT 100;

sched_start_interval_ms and off_cpu_gap_ms describe scheduling only. To measure mixer periods, use a confirmed application/platform slice emitted once per cycle, such as the AudioFlinger#MixerCycle event defined below. Order slices within each utid and calculate the difference between adjacent start timestamps. Do not compare arbitrary context switches with audio buffer deadlines.

After confirming that AudioFlinger#MixerCycle has been instrumented or mapped, calculate actual cycle start intervals:

WITH cycles AS (
  SELECT s.id, s.ts, s.dur, tt.utid,
    LEAD(s.ts) OVER (PARTITION BY tt.utid ORDER BY s.ts, s.id) AS next_ts
  FROM slice s
  JOIN thread_track tt ON s.track_id = tt.id
  WHERE s.name = 'AudioFlinger#MixerCycle' AND s.dur >= 0
    AND s.ts >= 120000000000 AND s.ts < 126000000000
    AND EXISTS (SELECT 1 FROM domain_objects d
      WHERE d.utid = tt.utid AND d.owner_confirmed = 1 AND d.role = 'audio_mixer')
)
SELECT utid, id AS slice_id, ts, dur / 1e6 AS cycle_work_ms,
  (next_ts - ts) / 1e6 AS mixer_cycle_period_ms
FROM cycles
WHERE next_ts IS NOT NULL
ORDER BY mixer_cycle_period_ms DESC;

Do not assign period_ms a fixed threshold across devices. It depends on sample rate, buffer size, fast path, Bluetooth route, offload, and how AAudio/OpenSL ES/AudioTrack is used. A more reliable method is to establish a baseline from clean playback on the same device and route, then compare whether the tail grows in the problematic capture, whether it grows across consecutive cycles, and whether that coincides with underrun logs or audible-problem timestamps.

Calculate Running and Runnable Separately

Many domain reports mix thread states together. Parts 09 and 11 already explain Running/Runnable definitions, the meaning of R+, and the prerequisites for scheduling evidence: complete sched data and no ftrace loss in stats. We will not repeat them here. Domain analysis differs in only one respect: the threads being measured come from the domain_objects dictionary with owner_confirmed = 1, rather than guesses based on thread names.

This SQL outputs both Running and Runnable for domain threads within the issue window:

WITH target_window AS (
  SELECT 120000000000 AS start_ts, 126000000000 AS end_ts
),
domain_threads AS (
  SELECT DISTINCT utid, thread_name, process_name
  FROM domain_objects
  WHERE owner_confirmed = 1
),
state_totals AS (
  SELECT s.utid,
    SUM(MIN(s.ts + s.dur, w.end_ts) - MAX(s.ts, w.start_ts)) AS covered_ns,
    SUM(CASE WHEN s.state = 'Running'
      THEN MIN(s.ts + s.dur, w.end_ts) - MAX(s.ts, w.start_ts) ELSE 0 END) AS running_ns,
    SUM(CASE WHEN s.state IN ('R', 'R+')
      THEN MIN(s.ts + s.dur, w.end_ts) - MAX(s.ts, w.start_ts) ELSE 0 END) AS runnable_ns
  FROM thread_state s
  CROSS JOIN target_window w
  WHERE s.dur > 0 AND s.ts < w.end_ts AND s.ts + s.dur > w.start_ts
  GROUP BY s.utid
)
SELECT
  t.utid, t.process_name, t.thread_name,
  CASE WHEN s.covered_ns = w.end_ts - w.start_ts THEN 'full'
       WHEN s.covered_ns IS NULL THEN 'missing' ELSE 'partial' END AS sched_coverage,
  CASE WHEN s.covered_ns = w.end_ts - w.start_ts THEN s.running_ns / 1e6 END AS running_ms,
  CASE WHEN s.covered_ns = w.end_ts - w.start_ts THEN s.runnable_ns / 1e6 END AS runnable_ms
FROM domain_threads t
CROSS JOIN target_window w
LEFT JOIN state_totals s USING (utid)
ORDER BY runnable_ms DESC, running_ms DESC;

This uses Running/Runnable states in thread_state for a consistent calculation. Durations remain blank when records are missing or cover only part of the window. full means only interval coverage and does not replace stats checks. If a thread was alive for only part of the window, narrow the window before calculating again.

If Runnable time is high, first examine CPU contention, thread priority, frequency, and wakeup sources. If Running time is high, consider CPU profiling or function-level hotspots.

Use Logs to Add Stage Semantics

Logs are useful for adding request IDs, session IDs, route changes, underruns, and error codes. When the trace includes android.log, restrict it to the issue window as well:

WITH target_window AS (
  SELECT 120e9 AS start_ts, 126e9 AS end_ts
)
SELECT
  l.ts / 1e9 AS ts_s,
  p.name AS process_name,
  l.prio,
  l.tag,
  l.msg
FROM android_logs l
LEFT JOIN thread th USING (utid)
LEFT JOIN process p USING (upid)
CROSS JOIN target_window w
WHERE l.ts >= w.start_ts
  AND l.ts < w.end_ts
  AND (
    LOWER(l.tag) LIKE '%camera%'
    OR LOWER(l.tag) LIKE '%audio%'
    OR LOWER(l.msg) LIKE '%underrun%'
    OR LOWER(l.msg) LIKE '%configurestream%'
    OR LOWER(l.msg) LIKE '%route%'
  )
ORDER BY l.ts;

Do not treat logs as the sole evidence. They name the stages; durations and blocking still need to be traced back to slices, sched, thread_state, Binder, and HAL threads.

Reports Must Lead Back to the UI

Automated reports should output more than SQL tables. Use four fixed sections:

Section	Output
Data quality	Trace duration, `stats` anomalies, presence of key data sources
Object dictionary	Relevant processes, threads, tracks, log tags, request/session IDs, confirmation status
Stage metrics	Camera open/config/HAL buffer/presented frame; Audio write/mix/output cycles
Review points	`ts`, `dur`, process, thread, slice/log names

An example structure:

Camera open analysis
trace: camera_open_run03.perfetto-trace
data_quality: clean
target_window: 123.000s..124.200s
phase:
  open_api: 42 ms
  configure_streams: 98 ms
  first_hal_buffer: 286 ms
  first_presented_frame: 486 ms
blocking:
  cameraserver Binder thread runnable: 41 ms
  provider HAL slice: 98 ms
review:
  ts=123.456s process=cameraserver thread=Binder:1234 slice=connectDevice dur=73ms
  ts=123.612s process=provider thread=CamX slice=configure_streams dur=98ms

Every anomaly in the report should lead back to Perfetto UI. Conclusions without timestamps cannot be reviewed by someone else.

For domain owners, a report must point to reviewable intervals rather than stop at “the script says Camera HAL is slow.” For example: provider configure took 98 ms, the request ID was 17, a Binder thread was Runnable for 41 ms at the same time, and there was no data loss. The script assembles the context; the final conclusion must still be verifiable against the UI and source code paths.

The server and case package also need a machine-readable summary.json:

{
  "case_id": "case-20260504-camera-001",
  "domain": "camera",
  "scenario": "camera_open_preview",
  "preset_id": "camera_open_lab_v4",
  "config_version": "v4",
  "trace_config_hash": "sha256:...",
  "perfetto_version": "v51",
  "trigger": "app_camera_open_slow",
  "target_window": {
    "start_ts": 123000000000,
    "end_ts": 124200000000,
    "source": "track_event"
  },
  "data_quality": "clean",
  "required_signals_available": true,
  "missing_required_signals": [],
  "stats_findings": [],
  "object_dictionary_version": "camera_domain_v1",
  "domain_objects": ["camera_app_client", "camera_service", "camera_provider"],
  "stage_metrics": [
    {
      "stage_name": "configure_streams",
      "status": "measured",
      "start_ts": 123612000000,
      "end_ts": 123710000000,
      "dur_ms": 98,
      "source_role": "camera_provider",
      "evidence_slice": "configure_streams",
      "review_ts": 123612000000
    }
  ],
  "blocking_evidence": [
    {
      "type": "runnable",
      "process": "cameraserver",
      "thread": "Binder:1234",
      "dur_ms": 41,
      "source_sql": "sql/domain_thread_state.sql"
    }
  ],
  "app_marker_status": "matched",
  "platform_event_status": "matched",
  "fallback_used": "none",
  "evidence_grade": "confirmed",
  "assignee_hint": "platform-camera",
  "ui_review_points": [
    {
      "ts": 123612000000,
      "dur": 98000000,
      "process": "provider",
      "thread": "CamX",
      "track_id": 88,
      "slice": "configure_streams"
    }
  ],
  "privacy_mode": "log_tag_whitelist",
  "redaction_status": "redacted",
  "raw_trace_available": true,
  "log_excerpt_policy": "window_only",
  "retention_until": "2026-05-07T15:30:12+08:00",
  "next_action": "owner_review"
}

Add Stable Events in the App and Platform

Name drift is a major problem for Camera/Audio automation. Slice names can vary across Android versions, vendor HALs, camera modules, and audio routes. Add stable events in code under your control:

Event	Owner	Required args	Purpose	Privacy boundary
`Camera#OpenStart`	App	`camera_id, session_id, operation_id, event_id`	Start of open_api	No user content
`Camera#OpenEnd`	App	`camera_id, session_id, operation_id, event_id, result`	End of open_api	Allowlisted error codes
`Camera#ConfigureStreamsStart`	Platform	`camera_id, session_id, operation_id, stream_id`	Start of configure stage	Resolution/format may be retained
`Camera#FirstPreviewFrame`	Platform/App	`camera_id, session_id, operation_id, stream_id, surface_id`	Candidate first visible frame	No image content
`Audio#TrackStart`	App/Platform	`track_id, route, sample_rate, buffer_frames`	Start of audio cycles	Route represented as an enum
`Audio#WriteBuffer`	App	`track_id, frames, buffer_level`	App write cadence	No media content
`Audio#Underrun`	Platform	`track_id, route, sample_rate, buffer_frames, underrun_count`	Underrun event	Allowlisted routes/tags
`Audio#RouteChange`	Platform	`old_route, new_route, reason`	Route change	No unique device identifiers

These events can come from android.os.Trace, NDK ATrace, Perfetto SDK Track Event, or platform ATRACE. Keep event names stable; do not concatenate dynamic IDs into them.

With ATrace, focus on stable slice names and async cookies. Prefer Track Event arguments, flow IDs, logs/side tables, or metadata in the same case for ID information. Counters should contain interpretable numeric values such as queue depth, buffer level, and underrun count. For richer arguments and cross-thread correlation, prefer Perfetto SDK Track Event as described in Part 13.

A platform-specific custom data source is a lower priority. Build one only for high-frequency, strongly structured data or information that ordinary slices/counters cannot represent. Its manifest must specify the data source name, producer process, SELinux domain, TraceConfig, parser, and SQL compatibility policy.

Expand Domain Methods into an Issue Checklist

Camera and Audio are only examples. To use Perfetto reliably as a team, translate common symptoms into capture presets, collectors, build types, analysis paths, and output evidence.

Issue	Base data sources	Additional domain signals	Default output
UI jank	sched, freq, idle, gfx, view, wm, input, process_stats, FrameTimeline	binder_driver, app ATrace/Track Event	Problematic frames, App/HWUI/SurfaceFlinger stages, thread states
Slow tap response	input, sched, freq, view, wm, binder_driver, FrameTimeline	App visual-state counters, high-speed camera comparison	Input to first visible change
App startup	sched, freq, am, wm, view, binder_driver, FrameTimeline	log, CPU profiling, app ATrace/Track Event	Startup stages, first frame, longest waits
ANR	sched, freq, binder_driver, am, wm, log	lock, IO, CPU profiling	Main-thread wait evidence
Slow Binder	sched, binder_driver, binder_lock, process_stats	`system_server` logs, interface events	Top N slow transactions, server state
Native memory	process_stats	heapprofd, log, application triggers	Allocation stacks and growth trends
Camera	sched, freq, camera, hal, binder_driver, FrameTimeline, log	CPU profiling, platform events	Open/config/HAL buffer/first visible frame
Audio	sched, freq, audio, hal, binder_driver, log	CPU profiling, platform events	Cycle jitter around underruns
Power/thermal management	sched, freq, idle, android.power, thermal, log	GPU counters, devfreq, power rails	Resource trends and anomalous windows

Every preset needs an owner, version, purpose, default window, collector, build boundaries, data sources, overhead level, privacy level, and quality gate. Without these fields, presets quickly become a pile of configurations that nobody dares to delete:

preset_id: ui_jank_v4
maintainer: platform-performance
mode: ring_buffer_stop_trigger
default_window: 30s
collector: platform_trace_service
consumer_identity: oem_perfetto_collector
selinux_domain: oem_tracing_service
allowed_data_sources:
  - linux.ftrace
  - android.surfaceflinger.frametimeline
  - track_event
can_read_trace_output: true
upload_path: oem_diagnostics
supported_builds:
  - userdebug
  - oem_diagnostics_user
app_marker_policy:
  atrace_apps:
    - com.example.app
  requires_profileable_or_debuggable: true
requires_userdebug_or_root: false
requires_vendor_producer: false
runtime_overhead_level: low
data_rate_budget_mb_s: 1.0
max_duration_s: 60
forbidden_sources_in_field:
  - linux.perf
  - heapprofd
  - gpu.counters
  - syscall_high_frequency
  - page_fault_high_frequency
privacy_level: system-metadata-and-app-markers
trace_config: configs/ui_jank_v4.pbtx
config_format: binary_proto
config_hash: sha256:...
review_status: approved
required_signals:
  - FrameTimeline
  - sched
  - thread_state
  - process_stats
fallback_sources:
  - bugreport_summary
  - external_logcat
quality_gate:
  quality_sql: sql/trace_health.sql
  disallow_data_loss: true
  required_stats_clean:
    - ftrace/sched
    - central_buffer
    - clock
  require_required_signals: true
report_template: reports/ui_jank.md

Ordinary apps cannot independently start system-level ftrace, process stats, or SurfaceFlinger FrameTimeline sessions. Apps can emit stable markers, send registered triggers, and provide local metadata. The system trace session is normally held by a platform service, OEM diagnostics component, Traceur, shell, or a laboratory script.

On user builds, only Traceur, shell, and platform-signed OEM components authorized by SELinux can hold system trace sessions. Arbitrary ordinary apps or unauthorized diagnostics apps cannot directly control the system-level Perfetto consumer socket. Production should accept only reviewed binary TraceConfigs or preset IDs, not arbitrary text protos.

App profiling on user builds also depends on profileable/debuggable. Before automatically uploading raw traces, logcat, or bugreports from production, there must be user authorization, an enterprise or OEM diagnostics agreement, a redaction policy, access controls, retention periods, and audit records. Configure trace_filter / field-level redaction on the capture or read side of production/bugreport paths, and record the filter-rule version in case metadata.

Reports also need evidence grades for their conclusions:

Grade	Wording	Required fields
Confirmed	`X blocked Y within window A`	`required_signals_available=true`, `data_quality=clean`, `ui_review_points>=1`, `source_sql`
Tentative	`The evidence points more toward X`	Primary evidence holds; `missing_evidence` lists missing source code, logs, or domain state
Not supported	`The evidence does not currently support X`	`signal_present=true` and metrics do not support that direction; this cannot be inferred from missing data
Downgraded	`This trace cannot determine X`	`degrade_reason` specifies data loss, a missing data source, or insufficient samples

These grades protect report quality. Many engineering disagreements stem not from the trace itself, but from assigning the wrong conclusion grade: evidence that supports only a tentative direction is presented as confirmed.

What the Platform and OEM Need to Build

Apps can add application semantics, but many issues occur outside their view: blocked system_server Binder calls, AudioFlinger underruns, Camera HAL blocking, Power HAL policy changes, scheduler CPU migrations, insufficient GPU execution tracks, or evidence scattered through a bugreport. The platform needs to standardize four things: presets, triggers, evidence-package format, and the automated analysis entry point.

Engineers should not have to write an ad hoc TraceConfig every time. Manage TraceConfig versions like an interface, with review, testing, and releases in a repository:

perfetto-presets/
  base/
    base_low_overhead_v1.pbtx
  ui/
    ui_jank_field_v3.pbtx
  media/
    audio_underrun_field_v3.pbtx
    camera_open_lab_v4.pbtx
  system/
    binder_slow_field_v2.pbtx
    thermal_degrade_field_v2.pbtx
    boottrace_v2.pbtx
  manifest.yaml
  sql/
    ui_jank_summary.sql
    audio_mixer_jitter.sql
    camera_open_stage.sql

A Perfetto infrastructure should have at least three layers:

Layer	Use case	Collector	Capture characteristics	Typical data sources
field	Production/staged-rollout issues on user builds	Platform/OEM diagnostics components or Traceur; apps emit only markers/triggers	Low overhead, ring buffer, stop trigger for a short window	sched, process stats, a few app ATrace events, necessary counters
lab	Laboratory reproduction and version comparisons	shell, test frameworks, platform scripts	Controlled devices, repeatable scripts, stable metrics	FrameTimeline, freq, idle, Binder, domain log excerpts
deep	Focused investigation	userdebug/root, platform engineering tools	Short duration, high overhead, manual operation	heapprofd, linux.perf, GPU counters, additional HAL events

Do not deploy deep presets in production. CPU profiling, native heap profiling, and GPU counters are valuable in laboratory and focused investigations, but are too expensive for long-running field capture on user builds and carry greater privacy and stability risks.

Document the boundaries of deep presets separately:

linux.perf is a sampling profiler, not a complete event stream. By default, restrict it to short windows, low sampling rates, and target processes/threads, and document unwind, kernel-frame, root/userdebug/kptr restrictions.
Before using vendor tracepoints, check /sys/kernel/tracing/events/*/*/format, field stability, and ftrace_setup_errors. Unknown events can serve only as raw events or candidate evidence.
The field layer prohibits linux.perf, heapprofd, GPU counters, and high-frequency syscall/pagefault events. Necessary counters must specify sampling period, units, supported devices, and upload privacy level.
Each preset needs measured write rates and a data-loss baseline. A cost_level is no substitute for a real budget.

A ring buffer is useful for retaining a short window around a trigger; it cannot preserve all context indefinitely. Long traces need the background capture, segmented storage, and explicit exit conditions from Part 15.

The trigger registry also needs consolidation. App agents, Framework watchdog, AudioFlinger, CameraService, Thermal/Power, and laboratory tools may send only registered trigger names. The platform then applies rate limits by device, user, version, scenario, network, and storage state. Once a name enters the platform, do not change it casually: doing so breaks server indexes, SQL templates, historical trends, and alert rules.

Integrate Bugreport with Perfetto as well. bugreport_score > 0 marks a running trace session as a Bugreport candidate; sessions with bugreport_score <= 0 are excluded. When Android dumpstate calls perfetto --save-for-bugreport, it selects the highest-scoring candidate trace and saves it to the Bugreport path. Android S/T saves the candidate trace and stops the original session early; Android U+ creates a read-only snapshot and lets the original session continue. bugreport_filename is an Android V / Perfetto v42+ field. bugreport_score > 0 also changes cloning/attachment behavior. Record supported versions, privacy level, and authorized access principals in the manifest.

Build a system instrumentation dictionary before adding code:

camera:
  CameraService#Open
  CameraProvider#ConfigureStreams
  CameraHAL#FirstRequest
  CameraHAL#FirstResult

audio:
  AudioTrack#Start
  AudioFlinger#MixerCycle
  AudioHAL#Write
  BluetoothAudio#RouteChange

This platform dictionary and the Camera#* names in the earlier domain schema are two distinct layers. CameraService#Open and CameraHAL#FirstResult are instrumentation implementation names inside individual services; Camera#OpenStart and Camera#FirstHalBuffer are interface names used for analysis and reporting. Match the layers through the owner field in the schema’s stable_events and a mapping table. SQL and reports use only schema names; renaming an implementation event requires updating only the mapping, not the analysis layer.

Names must be stable and their meaning suitable for SQL aggregation. Put dynamic information in arguments, counters, or metadata, rather than concatenating it into event names. Plaintext URLs, contacts, message content, search terms, and geographic locations must not enter trace events.

The server needs at least five functions: file acceptance checks, evidence-reliability checks, scenario summaries, aggregation indexes, and a manual review entry point. Before anyone opens the UI, the report should already identify the preset, trigger, trigger window, quality-gate status, relevant processes and threads, and the owner indicated by the machine-generated summary.

Represent processing status as an explicit state machine:

ingested
  -> quality_checked
  -> scenario_classified
  -> assignee_selected
  -> evidence_graded
  -> issue_linked
  -> fix_build_recorded
  -> same_preset_rerun
  -> before_after_compared
  -> regression_rule_updated

This state machine constrains the provenance of evidence: every conclusion can be traced back to the same preset, SQL, and evidence grade. After a version upgrade, renamed fields, missing events, or unavailable data sources can trigger a downgrade directly at quality_checked, preventing reports from continuing to emit apparently certain conclusions.

Common Pitfalls

Matching only LIKE '%camera%' misses providers, vendor HALs, codecs, and media processes.
Enabling only atrace_categories misses app-side android.os.Trace; use atrace_apps or Track Event.
Ordinary apps cannot enable system ftrace/process stats themselves; they can only cooperate with a platform collector by emitting markers or triggers.
FrameTimeline is not optional for UI, startup, or tap-response analysis. Without it, downgrade to thread/stage analysis.
Design for the documented userdebug support of android.log. On rooted user devices, verify permissions and actual data; prepare alternative sources for ordinary user builds.
Slow Binder is a symptom. The server may be waiting for HAL, a lock, IO, or downstream hardware.
Analyze Audio cycle jitter rather than only the longest slices.
Separate Camera open, configure, preview, and capture rather than combining them into one duration.
For Running, use actual execution intervals (sched or Running in thread_state); for Runnable, use R/R+ in thread_state. Calculate them separately.
An agent or script organizes evidence; it cannot replace UI review and domain-owner judgment.
Distinguish API return, HAL buffer return, and visible presentation when measuring the first camera frame.
Establish Audio period thresholds against a baseline for the device, route, and buffer configuration.
Increasingly heavy production presets create power, privacy, and stability problems.
Without a system instrumentation dictionary, event-name drift invalidates SQL and historical trends.
Keeping Bugreports and traces separate forces engineers to locate the same time window manually across multiple attachments.

Summary

For domain problems such as Camera and Audio, standardize a preset, build an object dictionary, write SQL around stages and cycles, produce reports with timestamps, and return to the UI to review anomalies.

The same approach applies to WebView, Flutter, games, and vendor-defined system services. Only the object dictionary and metric templates change. The foundation remains trace quality, process/thread ownership, stage durations, scheduling states, and log semantics. Add platform support for presets, triggers, Bugreport, system instrumentation, server summaries, and permission boundaries, and Perfetto can grow from an individual tool into a team capability.

You do not need to build this engineering workflow from scratch. My open-source project SmartPerfetto has already implemented much of it: scenario presets, SQL skills, evidence rules, and an AI agent runtime form a reusable automated trace-analysis platform. Object dictionaries, evidence grading, and conclusions that can be reviewed in the UI are built-in constraints. For the current version’s full capabilities, see SmartPerfetto: A Six-Week Update Review.