What is Grafana Tempo? Meaning, Examples, Use Cases & Complete Guide?

By /

Quick Definition

Grafana Tempo is an open source, distributed tracing backend designed for storing, querying, and reconstructing traces at scale for cloud-native applications.

Analogy: Tempo is like a flight recorder for distributed systems—collecting traces from many services so you can replay the sequence of calls that led to a failure.

Formal technical line: Tempo ingests spans, stores them as compressed trace chunks indexed with minimal metadata, and reconstructs full traces on demand for trace queries and correlation with logs and metrics.

If Grafana Tempo has multiple meanings:

  • Most common: Distributed tracing backend in the Grafana observability stack.
  • Also used to refer to: The Tempo project repository or Tempo deployment when teams shorthand it in ops chat.
  • Not commonly used to mean any Grafana frontend component.

What is Grafana Tempo?

Explain:

  • What it is / what it is NOT
  • Key properties and constraints
  • Where it fits in modern cloud/SRE workflows
  • A text-only “diagram description” readers can visualize

What it is:

  • A horizontally scalable, mostly index-light distributed tracing backend optimized for high volume.
  • Designed to integrate with OpenTelemetry, Jaeger, Zipkin, and Tempo native clients.
  • Focuses on storing spans efficiently and providing trace retrieval using external indexes like traces by trace ID, traces from logs, or by service name.

What it is NOT:

  • Tempo is not a metrics store; it does not replace a TSDB.
  • Tempo is not a full log storage solution; it focuses on trace spans and links to logs rather than storing raw log lines.
  • Tempo is not a visualization UI; it integrates with Grafana for trace views.

Key properties and constraints:

  • Index-light by design to reduce storage cost, relying on trace IDs and optional minimal indexes.
  • Scales horizontally with compactor and querier components.
  • Works best when paired with a metrics system and log storage for full observability.
  • Retention and compaction policies affect query latency and retrieval completeness.
  • Ingress throughput often limited by underlying object storage and network IOPS.

Where it fits in modern cloud/SRE workflows:

  • Complements metrics (for SLIs/SLOs) and logs (for payloads) by providing call-level causality.
  • Used during incident response to trace request lifecycles and pinpoint latency or error propagation.
  • Integrated into CI/CD to validate tracing coverage and detect regressions in latency.

Diagram description (text-only):

  • Instrumented app -> Collector (OTel/agent) -> Tempo Ingester -> Writes trace chunks to object storage -> Compactor consolidates chunks and uploads index artifacts -> Querier pulls chunks from object storage to reconstruct traces -> Grafana UI queries the Querier and links to logs and metrics.

Grafana Tempo in one sentence

Grafana Tempo is a scalable, cost-efficient tracing backend that stores and retrieves distributed traces while minimizing index overhead, intended to be used alongside metrics and logs for end-to-end observability.

Grafana Tempo vs related terms (TABLE REQUIRED)

ID Term How it differs from Grafana Tempo Common confusion
T1 Jaeger Jaeger is a tracing system with its own storage backends and UI People confuse UI with backend
T2 Zipkin Zipkin is a tracing format and light server for traces Confusion over protocol vs storage
T3 OpenTelemetry OpenTelemetry is a collection of APIs and SDKs for telemetry Often used interchangeably with backends
T4 Prometheus Prometheus stores metrics not traces Confusion about metrics vs traces
T5 Loki Loki stores logs and indexes them by labels Confusion about Tempo linking to logs
T6 Tempo Querier Component that reconstructs traces from storage People call the whole system Querier
T7 Tempo Compactor Component that batches and compacts trace chunks Confused with ingester behavior
T8 Object storage Durable blob store used by Tempo for chunks Thought of as optional local DB
T9 Trace sampling Sampling reduces volume at instrumentation time Confused with retention or compaction

Row Details (only if any cell says “See details below”)

  • None

Why does Grafana Tempo matter?

Cover:

  • Business impact (revenue, trust, risk)
  • Engineering impact (incident reduction, velocity)
  • SRE framing (SLIs/SLOs/error budgets/toil/on-call) where applicable
  • 3–5 realistic “what breaks in production” examples

Business impact

  • Faster root cause identification reduces mean time to resolution, which can lower revenue loss during outages.
  • By enabling precise attribution of failures to services or deployments, Tempo helps maintain customer trust and reduce SLA violations.
  • Temporal visibility into call paths reduces risk in complex microservice environments.

Engineering impact

  • Developers can validate end-to-end request flows and debug regression-induced latency.
  • Observability-driven development improves deployment confidence and accelerates delivery velocity.
  • Reduces firefighting toil by enabling targeted fixes rather than broad rollbacks.

SRE framing

  • SLIs: request latency percentiles, error-rate tied to traces with error spans.
  • SLOs: use Tempo traces to validate that 99th percentile request latency meets objectives.
  • Error budgets: trace-derived evidence helps determine whether errors are systemic or transient.
  • Toil and on-call: detailed traces reduce time spent reproducing incidents, lowering on-call stress.

What commonly breaks in production

1) Cascading retries causing amplified latency and resource exhaustion. 2) A third-party API call becoming slow, causing backend threads to block. 3) Bad serialization causing high CPU on a service and elevated tail latencies. 4) Misconfigured sampling causing missing traces for important endpoints. 5) Object storage misconfiguration causing Tempo queries to fail or return partial traces.

Where is Grafana Tempo used? (TABLE REQUIRED)

Explain usage across architecture, cloud, ops layers.

ID Layer/Area How Grafana Tempo appears Typical telemetry Common tools
L1 Edge and ingress Traces start at API gateway and include client metadata HTTP trace spans and headers Envoy, Nginx, Istio
L2 Service mesh Traces propagate via sidecars with context RPC and network spans Istio, Linkerd
L3 Application services Instrumented code emits spans and attributes Span durations and tags OpenTelemetry SDKs
L4 Database and cache DB spans show query durations and errors DB query spans and rows Postgres, Redis clients
L5 Background jobs Worker traces link to async flows Job start and completion spans Celery, Sidekiq
L6 Serverless / PaaS Traces include function invocations and cold starts Short-lived invocation spans Lambda, Cloud Functions
L7 Cloud infra Traces correlate with infra events and autoscaling Instance and container lifecycle spans Kubernetes, cloud APIs
L8 CI/CD pipelines Traces validate deployment path and failures Build and deploy step spans GitOps tools, CI runners
L9 Security monitoring Trace anomalies indicate chaining attacks Suspicious call patterns and timings SIEMs, security agents
L10 Incident response Tempo used to reconstruct fault paths Error spans and causal chains Pager duty, runbooks

Row Details (only if needed)

  • None

When should you use Grafana Tempo?

Include:

  • When it’s necessary
  • When it’s optional
  • When NOT to use / overuse it
  • Decision checklist
  • Maturity ladder
  • Examples for small teams and large enterprises

When necessary

  • You have a distributed system where requests cross multiple services and you need causality to troubleshoot.
  • You have frequent production incidents where pinpointing the service call chain reduces MTTR.
  • You need to correlate traces with logs and metrics to verify SLO breaches.

When optional

  • Monolithic apps where in-process profiling and logs suffice for debugging.
  • Systems where latency is negligible and error causes are isolated to a single service.

When NOT to use / overuse

  • Avoid tracing every internal diagnostic event; excessive spans increase cost and noise.
  • Don’t rely on tracing as the only observability source; combine with metrics and logs.
  • Avoid full tracing retention at high cardinality without cost controls.

Decision checklist

  • If requests cross 3+ services and you see production incidents -> use Tempo.
  • If your error budget is tight and SLOs require root cause clarity -> use Tempo.
  • If trace volume is extremely high and budget constrained -> consider selective sampling and aggregation.

Maturity ladder

  • Beginner: Instrument critical paths, use default sampling 1-10%, basic Grafana trace panels.
  • Intermediate: Add trace-to-log linking, tail-based sampling, and SLOs informed by traces.
  • Advanced: Full span enrichment, dynamic sampling, adaptive retention, and automated tracing-driven remediation.

Example decision: small team

  • Small indie app with 3 services, limited budget: Instrument core user flows, 5% sampling, store 7–14 days.

Example decision: large enterprise

  • Large microservices platform: Instrument all services, tail-based sampling to retain error and high-latency traces, integrate with centralized object storage and long-term retention for audits.

How does Grafana Tempo work?

Explain step-by-step:

  • Components and workflow
  • Data flow and lifecycle
  • Edge cases and failure modes
  • Short examples/pseudocode

Components and workflow

  • Instrumentation: Apps emit spans via OpenTelemetry or other clients.
  • Collector/Agent: Receives spans, optionally batches and exports to Tempo ingesters.
  • Ingester: Accepts spans, assembles into chunks, and uploads to object storage.
  • Object Storage: Durable blob store for chunks and artifacts.
  • Compactor: Merges and compacts chunks, builds auxiliary indexes if configured.
  • Querier: Retrieves chunks and reconstructs traces for query requests.
  • Distributor and Ruler: Coordinates write paths and sampling (if present).
  • Grafana: UI to query Tempo and visualize traces, and to correlate with logs/metrics.

Data flow and lifecycle

1) App emits spans with trace ID and parent ID. 2) Collector forwards spans to Tempo ingesters. 3) Ingester groups spans into chunks by trace ID and uploads to object storage. 4) Compactor consolidates chunks to improve retrieval and optionally builds indexes. 5) On query, the Querier fetches relevant chunks from object storage and reassembles the trace for display.

Edge cases and failure modes

  • Partial traces: If some spans are lost due to sampling or network failure, traces may appear incomplete.
  • Cold reads: Fetching from deep object storage can add latency.
  • Index inconsistency: If auxiliary indexes are missing or delayed, lookup by metadata can fail.
  • Object storage throttling: High throughput can cause write failures or slow reads.

Short pseudocode example (conceptual)

  • Instrumentation:
  • Start span on request entry
  • Add service and route tags
  • End/span on response or error
  • Collector config outline:
  • Receive spans -> batch by 1s or 1000 spans -> export to tempo endpoint
  • Query:
  • Grafana sends trace ID -> Querier fetches chunks -> reconstruct -> present

Typical architecture patterns for Grafana Tempo

1) Minimal Tempo with object store – Use case: cost-sensitive, low indexing needs – When: small to medium deployments 2) Tempo with trace-to-log linking – Use case: rapid developer debugging – When: teams have centralized log storage 3) Tempo with tail-based sampling and dynamic rules – Use case: high-volume platforms – When: need to preserve error/tail traces 4) Tempo in service mesh environment – Use case: automatic distributed context propagation – When: using Istio or Linkerd 5) Managed Tempo or SaaS tracing – Use case: offload maintenance – When: teams prefer managed observability

Failure modes & mitigation (TABLE REQUIRED)

ID Failure mode Symptom Likely cause Mitigation Observability signal
F1 Partial trace fetch Trace missing spans Span loss at ingestion Verify collector and retry logic Missing parent spans
F2 High query latency Slow trace load Cold object storage reads Enable compaction and local cache Elevated trace response time
F3 Elevated storage cost Unexpected bill increase High retention or no sampling Implement tail sampling and retention Rising storage usage metric
F4 Object write errors Ingest failures Object store throttling or creds Add retries and backoff, check permissions Ingest error rate
F5 Incorrect trace context Broken parent-child links Missing context propagation Fix instrumentation or headers Orphan spans without parents
F6 Index inconsistency Lookup by service fails Compactor lag or config error Re-run compaction and check logs Failed index lookup events
F7 Overloaded ingesters Rejected spans Insufficient capacity Scale ingesters or throttle clients High ingestion error counters

Row Details (only if needed)

  • None

Key Concepts, Keywords & Terminology for Grafana Tempo

Create a glossary of 40+ terms:

  • Term — 1–2 line definition — why it matters — common pitfall

Trace — A single end-to-end request journey across services — Shows causality for a user request — Pitfall: missing spans due to sampling. Span — A timed operation within a trace — Building block of traces for timing and metadata — Pitfall: overly granular spans create noise. Trace ID — Unique identifier for a trace — Essential for reconstructing traces — Pitfall: not propagated across services. Parent ID — Identifier of the parent span — Builds hierarchical call relationships — Pitfall: lost parent ID breaks trace chains. OpenTelemetry — Telemetry standard for traces, metrics, logs — Widely supported for instrumentation — Pitfall: SDK misconfiguration causes missing attributes. Collector — Component that receives telemetry and forwards it — Centralizes batching and export logic — Pitfall: wrong exporter config drops data. Ingester — Tempo component that accepts spans and writes chunks — Responsible for constructing trace blobs — Pitfall: insufficient ingester capacity. Chunk — Compressed group of spans stored as an object — Reduces storage overhead and IO — Pitfall: large chunks increase read latency. Compactor — Merges trace chunks and prepares indexes — Improves read efficiency and reduces storage fragmentation — Pitfall: misconfigured compaction leads to stale indexes. Querier — Component that retrieves and reconstructs traces for queries — Handles ray of retrieval from storage — Pitfall: querier overload causes slow trace page loads. Distributor — Component that partitions incoming writes to ingesters — Balances write load — Pitfall: misrouting can cause uneven ingestion. Object storage — Blob store used for durable trace chunk retention — Cost-effective long-term storage — Pitfall: slow storage yields high read latency. Index-light — Design approach minimizing auxiliary indexes to save cost — Reduces storage and ingestion cost — Pitfall: slower queries for metadata searches. Tail-based sampling — Sampling strategy that retains traces with errors or high latency — Preserves meaningful traces — Pitfall: complexity and late decision making. Head-based sampling — Sampling at request start based on rules — Simpler and lower overhead — Pitfall: may drop error traces. Trace enrichment — Adding useful attributes like user id or route to spans — Makes traces actionable — Pitfall: high cardinality attributes increase costs. Retention policy — How long traces are kept — Balances cost vs forensic needs — Pitfall: too short retention hinders postmortem. Trace ID lookup — Query by trace ID to retrieve a trace — Fast direct retrieval mechanism — Pitfall: requires instrumented clients emitting trace IDs. Trace view — UI presentation of reconstructed spans and timing — Facilitates human debugging — Pitfall: heavy traces are hard to scan. Sampling rate — Fraction of traces to keep — Controls volume and cost — Pitfall: incorrect rates lose signal. Service map — Visualization of service interactions from traces — Highlights call topology — Pitfall: map noisiness from transient calls. Span attributes — Key-value pairs attached to spans — Useful for filtering and drilldown — Pitfall: PII or sensitive data in attributes. Latency percentiles — Quantiles of span durations like p50, p95, p99 — Used for SLO evaluation — Pitfall: relying on averages conceals tail latency. SLO (tracing-informed) — Objective defined using trace-derived SLIs like p95 latency — Ties tracing to reliability targets — Pitfall: too many SLOs dilute focus. SLI (trace-based) — Observable indicator derived from traces like error rate of traces — Directly measures user-facing outcomes — Pitfall: incomplete instrumentation skews SLIs. Burn rate — Rate at which error budget is consumed — Guides escalation and throttling — Pitfall: reactive alerts without burn awareness. Correlation ID — Client-supplied ID to correlate logs and traces — Helps join different telemetry types — Pitfall: inconsistent usage across services. Trace reconstruction — Process of assembling spans into full trace — Core function of Tempo — Pitfall: missing chunks cause partial traces. Chunk compaction delay — Time between initial write and compaction — Affects query freshness — Pitfall: long delays hinder immediate debugging. Retention tiering — Store different retention lengths for hot vs cold traces — Optimizes cost — Pitfall: complex retrieval paths. Cold read — Fetch from deep storage that increases latency — Expected for long-retained traces — Pitfall: user-facing slow trace loads. Hot cache — Local cache to speed queries for recent traces — Improves UX — Pitfall: stale cache returns old data. Span compression — Reducing size of spans before storage — Saves cost and bandwidth — Pitfall: decompress costs on read. Metadata index — Small index used for service or operation lookups — Speeds metadata queries — Pitfall: high cardinality metadata spikes index size. Trace sampling bias — Sampling that skews dataset representation — Affects statistical conclusions — Pitfall: non-random sampling biases SLO measures. Instrumentation library — SDKs used to create spans in code — Enables consistent tracing — Pitfall: mixing SDKs without context propagation. Distributed context propagation — Passing trace context across process boundaries — Critical for end-to-end tracing — Pitfall: missing headers in async calls. Adaptive sampling — Dynamic sampling based on traffic and errors — Saves cost while preserving signal — Pitfall: complexity in configuration. Quota management — Limits to control ingestion and costs — Prevents runaway usage — Pitfall: hard limits can drop critical telemetry. Observability pipeline — Flow from instrumentation to storage to UI — Provides end-to-end visibility — Pitfall: pipeline gaps create blind spots. Trace correlation — Linking traces to logs and metrics via IDs — Accelerates root cause analysis — Pitfall: missing correlation keys. Long tail traces — Low-frequency but high-impact traces like rare errors — Important for incident analysis — Pitfall: often lost to sampling. Synchronous vs asynchronous export — Exporting spans inline vs background — Tradeoff of latency vs reliability — Pitfall: synchronous export may increase request latency. Query federation — Combining trace queries with logs and metrics at UI time — Enables contextual debugging — Pitfall: cross-store latency. Cost per trace — Economic model for each retained trace — Drives sampling and retention choices — Pitfall: unmonitored growth leads to surprises. Grafana integration — Linking Tempo to Grafana for visualization and alerting — Standard UX for many teams — Pitfall: mismatched Grafana versions cause compatibility issues.

How to Measure Grafana Tempo (Metrics, SLIs, SLOs) (TABLE REQUIRED)

Must be practical:

  • Recommended SLIs and how to compute them
  • “Typical starting point” SLO guidance
  • Error budget + alerting strategy
ID Metric/SLI What it tells you How to measure Starting target Gotchas
M1 Trace ingest rate Volume of spans or traces per second Instrument ingesters and collectors counters Varies by org Spikes may be sampling config issues
M2 Trace query latency Time to reconstruct a trace Measure Querier response times p95 < 1s for hot reads Cold reads are slower
M3 Trace completeness Percent traces with full span set Compare known requests vs reconstructed Aim for >95% for critical flows Sampling reduces completeness
M4 Ingest error rate Failed spans accepted by Tempo Count export failures and 5xx responses <0.1% typical Network errors inflate this
M5 Storage growth rate Increase in object storage size per day Monitor storage metrics and bills Keep growth predictable Sudden growth indicates missing sampling
M6 Trace retention success Percent traces retrievable by age Periodic retrieval checks 100% within retention window Compaction errors may lose chunks
M7 Tail latency SLI Fraction of requests under latency threshold Use traces to compute p99 or p95 success p95 or p99 target per service Dependent on instrumentation accuracy
M8 Error-trace capture rate Fraction of error traces preserved Tail-based sampling metrics Aim >95% for errors Head sampling may drop errors
M9 Compaction lag Time between upload and compaction complete Monitor compactor metrics Keep under minutes for hot data Long lags hurt queries
M10 Cost per trace Monetary cost allocated per trace/day Billing divided by trace volume Compute internally Variable by storage and egress

Row Details (only if needed)

  • None

Best tools to measure Grafana Tempo

Pick 5–10 tools. For each tool use this exact structure (NOT a table):

Tool — Prometheus

  • What it measures for Grafana Tempo: Ingest rates, error counters, component latencies, compactor metrics.
  • Best-fit environment: Self-hosted Kubernetes and cloud VM clusters.
  • Setup outline:
  • Scrape Tempo component metrics endpoints.
  • Define recording rules for p95/p99.
  • Export alerts to Alertmanager.
  • Correlate with Grafana dashboards.
  • Strengths:
  • Established open source metrics ecosystem.
  • Fast querying for time series.
  • Limitations:
  • Not a tracing store; correlates only via labels.
  • Requires managing Prometheus at scale.

Tool — Grafana

  • What it measures for Grafana Tempo: Visualization of traces, dashboards combining traces/logs/metrics.
  • Best-fit environment: Teams using Grafana for observability.
  • Setup outline:
  • Add Tempo as a data source.
  • Build dashboards for ingest and query latency.
  • Configure panel links between logs and traces.
  • Strengths:
  • Unified UX for traces and metrics.
  • Powerful alerting integrations.
  • Limitations:
  • Visualization only; storage elsewhere.
  • Requires correct data source plugin versions.

Tool — OpenTelemetry Collector

  • What it measures for Grafana Tempo: Provides export telemetry metrics and health; controls sampling.
  • Best-fit environment: Any instrumented apps using OpenTelemetry.
  • Setup outline:
  • Deploy collector as agent or gateway.
  • Configure exporters to Tempo.
  • Add processors for batching and sampling.
  • Strengths:
  • Flexible pipeline for telemetry.
  • Consistent instrumentation across languages.
  • Limitations:
  • Complexity in pipeline configs.
  • Resource usage when deployed as agents.

Tool — Object storage metrics (S3/GCS)

  • What it measures for Grafana Tempo: Storage usage, egress, request latency and error rates.
  • Best-fit environment: Tempo backed by cloud object storage.
  • Setup outline:
  • Enable storage metrics from provider.
  • Monitor per-bucket metrics and alerts.
  • Correlate with Tempo ingest spikes.
  • Strengths:
  • Direct view into storage cost drivers.
  • Alerts on throttling and errors.
  • Limitations:
  • Provider-specific metric names.
  • Some metrics may be delayed.

Tool — CI/CD pipelines (build integration)

  • What it measures for Grafana Tempo: Trace coverage per deployment and instrumentation regressions.
  • Best-fit environment: Teams practicing observability-as-code.
  • Setup outline:
  • Run smoke requests during deploy and validate trace presence.
  • Fail pipelines on missing critical traces.
  • Include trace-based SLO checks.
  • Strengths:
  • Prevents regressions of telemetry.
  • Automates basic reliability checks.
  • Limitations:
  • Requires test harness and synthetic traffic.

Recommended dashboards & alerts for Grafana Tempo

Executive dashboard

  • Panels: Total traces per minute, ingest error rate, storage growth, average query latency.
  • Why: High-level health and cost visibility for stakeholders.

On-call dashboard

  • Panels: Current active incidents, top error traces, slowest traces, recent compactor lags, ingester health.
  • Why: Rapid diagnostics for on-call engineers.

Debug dashboard

  • Panels: Trace search by service, detailed trace waterfall, span duration histogram, trace-to-log join panels.
  • Why: Deep dive tools for debugging specific requests.

Alerting guidance

  • Page vs ticket:
  • Page when SLO burn rate exceeds threshold or when trace ingestion fails globally.
  • Ticket for sustained storage growth or low-priority compactor lag.
  • Burn-rate guidance:
  • Use 3x or 4x burn rate thresholds to trigger escalation as error budget consumption accelerates.
  • Noise reduction:
  • Deduplicate alerts using grouping keys like service and operation.
  • Suppress noisy alerts during deployments via maintenance windows.
  • Use rate-limited alerting to avoid paging for transient spikes.

Implementation Guide (Step-by-step)

Provide:

1) Prerequisites 2) Instrumentation plan 3) Data collection 4) SLO design 5) Dashboards 6) Alerts & routing 7) Runbooks & automation 8) Validation (load/chaos/game days) 9) Continuous improvement

1) Prerequisites – Inventory services and critical application paths. – Choose object storage and provisioning for Tempo (S3-compatible, GCS, or Azure blob). – Ensure Grafana or compatible UI available. – Decide sampling strategy and retention policy.

2) Instrumentation plan – Adopt OpenTelemetry SDKs where possible. – Define standard span naming and attributes for service, route, error flag, and user id (if compliant). – Instrument critical business transactions first. – Implement context propagation across sync and async boundaries.

3) Data collection – Deploy OpenTelemetry Collector as sidecar or gateway. – Configure batching, retry, and exporter to Tempo with TLS and authentication. – Enable service-level sampling and tail-based sampling processors if needed.

4) SLO design – Define SLIs using trace-derived metrics such as p95 latency for checkout API. – Set SLO targets and error budgets with stakeholders. – Map SLOs to alert burn rates.

5) Dashboards – Build executive, on-call, and debug dashboards described above. – Create per-service dashboards showing trace volumes and tail latencies.

6) Alerts & routing – Define alert severity based on SLO burnout and ingestion availability. – Route pages to on-call teams and tickets to platform or reliability engineers. – Implement suppression during planned releases.

7) Runbooks & automation – Create runbooks for common issues: missing traces, ingestion errors, compactor backlog. – Automate self-healing where safe (scale ingesters, restart failed collectors). – Implement CI checks for instrumentation coverage.

8) Validation – Load testing: validate ingestion and query latency at production-like volume. – Chaos testing: simulate object store latency and network partitions. – Game days: run incident scenarios to test runbooks and alerting.

9) Continuous improvement – Weekly review of top error traces and root cause trends. – Monthly sampling and retention tuning based on cost and signal. – Quarterly instrument expansion and SLO reviews.

Checklists

Pre-production checklist

  • Instrument at least 80% of critical paths.
  • Collector configured and tested with Tempo endpoint.
  • Object storage credentials and permissions validated.
  • Baseline dashboards and alerts created.
  • Load test to simulate expected traffic.

Production readiness checklist

  • Alerting routes and escalation policies in place.
  • Retention and sampling policies configured and documented.
  • Compactor and querier scaling validated.
  • Cost thresholds and quotas configured.
  • Runbooks published and on-call trained.

Incident checklist specific to Grafana Tempo

  • Verify collectors and ingesters are healthy.
  • Check object storage write and read metrics.
  • Confirm compactor is running and MBI queues are clear.
  • Validate trace IDs for recent incidents are retrievable.
  • If data missing, check sampling rules and exporter errors.

Kubernetes example

  • Deploy OpenTelemetry collector as DaemonSet.
  • Configure Tempo as backend in collector exporter section.
  • Use Kubernetes resource requests to ensure collector stability.
  • Verify with synthetic traces using a test pod.

Managed cloud service example

  • For managed PaaS, configure tracing SDKs with exporter endpoint provided by Tempo SaaS or managed instance.
  • Use cloud provider object storage for Tempo backing store.
  • Validate IAM roles and object store ACLs for writes and reads.

Use Cases of Grafana Tempo

Provide 8–12 use cases:

  • Context
  • Problem
  • Why Grafana Tempo helps
  • What to measure
  • Typical tools

1) Slow checkout in e-commerce – Context: High-value checkout with multiple microservices. – Problem: Intermittent high latency at p99 causing abandoned carts. – Why Tempo helps: Reveals exact service or DB call causing the tail latency. – What to measure: p95/p99 latency per service, error traces captured. – Typical tools: Tempo, Prometheus, Grafana, Postgres traces.

2) Third-party API degradation – Context: Payments gateway used by checkout service. – Problem: External API slowdowns propagate to users. – Why Tempo helps: Shows call chain delays and retry amplification. – What to measure: External call duration distribution, retry counts. – Typical tools: Tempo, OpenTelemetry, CI synthetic tests.

3) Microservice dependency mapping – Context: Rapidly changing service topology. – Problem: Unknown dependencies causing deployment risk. – Why Tempo helps: Service map from traces shows callers and callees. – What to measure: Call frequency, error rate per dependency. – Typical tools: Tempo, Grafana service map, service mesh telemetry.

4) Debugging serverless cold starts – Context: Function-based architecture with variable cold starts. – Problem: Cold start spikes affect latency for certain endpoints. – Why Tempo helps: Captures invocation span with cold start markers. – What to measure: Invocation latency, cold start occurrence. – Typical tools: Tempo, function tracing SDK, logs.

5) Database performance regression – Context: New ORM change introduced slow queries. – Problem: CPU and latency spikes correlated with specific SQL. – Why Tempo helps: DB spans show query durations and callers. – What to measure: Query latency by statement, number of rows. – Typical tools: Tempo, DB monitoring tools, tracing-enabled DB client.

6) Background job failures – Context: Batches processed asynchronously with retries. – Problem: Jobs silently failing or retry storms. – Why Tempo helps: Traces for job lifecycle detect failure points and retry chains. – What to measure: Job success rate, retry counts, processing time. – Typical tools: Tempo, job system metrics, centralized logs.

7) Deployment verification – Context: Continuous delivery pipeline rolling out new versions. – Problem: Telemetry regressions post-deploy not caught. – Why Tempo helps: CI can validate traces for critical flows before and after deploy. – What to measure: Feature flow latency before/after deploy, trace coverage. – Typical tools: Tempo, CI pipelines, synthetic testers.

8) Security investigation – Context: Detecting anomalous call chains indicating abuse. – Problem: Multi-step misuse across services. – Why Tempo helps: Trace patterns reveal the sequence and services involved. – What to measure: Unusual high-frequency traces, unusual attribute values. – Typical tools: Tempo, SIEM, log correlation.

9) Autoscaling tuning – Context: Autoscaler reacts to metrics but not to call depth. – Problem: Thundering herd or inadequate scale for tail latency. – Why Tempo helps: Trace depth and queue times indicate real concurrency pressure. – What to measure: Concurrent calls per service, queue wait times. – Typical tools: Tempo, metrics system, autoscaler signals.

10) Postmortem evidence collection – Context: Incident requires forensic analysis for compliance. – Problem: Need durable evidence of calls and timings. – Why Tempo helps: Retained traces provide sequence and payload references. – What to measure: Trace retention completeness and metadata. – Typical tools: Tempo, long-term object storage, audit logs.

Scenario Examples (Realistic, End-to-End)

Scenario #1 — Kubernetes microservice regression

Context: A Kubernetes cluster runs 20 microservices behind a service mesh. After a release, p99 latency increases for the API gateway. Goal: Identify the service and code path causing p99 latency. Why Grafana Tempo matters here: Tempo reconstructs request paths across pods and shows which service spans dominate tail latency. Architecture / workflow: Istio sidecars propagate context -> Services emit spans to OTel Collector DaemonSet -> Collector exports to Tempo ingesters -> Chunks to object storage -> Grafana queries. Step-by-step implementation:

1) Ensure OpenTelemetry SDKs instrument services. 2) Deploy Collector as DaemonSet with Tempo exporter. 3) Configure a tail-based sampling rule to keep error and high-latency traces. 4) Deploy Tempo with S3 backed storage and compactor. 5) Create Grafana dashboards focusing on p95/p99 by route. 6) Trigger test load and compare traces pre and post release. What to measure: p99 latency per route and service, number of spans per trace, compactor lag. Tools to use and why: Tempo for traces, Prometheus for metrics, Grafana for visualization, Istio for context propagation. Common pitfalls: Missing instrumentation in some services, sidecar config blocking headers. Validation: Reproduce the regression in a staging load test and verify traces expose the problematic DB call. Outcome: Root cause found in an inefficient query inside service B; patch reduced p99 by 40%.

Scenario #2 — Serverless payment flow slowdowns

Context: A managed cloud PaaS using functions for payment processing reports sporadic user checkout timeouts. Goal: Detect cold start and downstream call latency causing timeouts. Why Grafana Tempo matters here: Tempo captures function invocation spans and downstream service spans across provider boundaries. Architecture / workflow: Function instrumented with OTel -> Exporter sends to a managed collector in cloud -> Tempo ingests to managed Tempo instance -> Grafana links traces to logs. Step-by-step implementation:

1) Add OTel SDK to functions and export to collector. 2) Ensure sampling preserves error traces. 3) Instrument third-party gateway calls and DB calls for context. 4) Create alert for p99 invocation latency. What to measure: Invocation latency distribution, cold start flag presence, downstream call latency. Tools to use and why: Tempo, managed collector, application logs. Common pitfalls: Missing propagation when calling external APIs, billing for high cold read rates. Validation: Synthetic load simulating scaled cold starts and validating trace capture. Outcome: Cold-start spikes identified and mitigated using warmers and reduced timeouts.

Scenario #3 — Incident response and postmortem

Context: Production outage with intermittent 5xx errors across services during peak traffic. Goal: Rapidly pinpoint cascade origin and create postmortem evidence. Why Grafana Tempo matters here: Traces show the sequence of failed calls and time ordering for postmortem. Architecture / workflow: Apps send spans to Tempo; on-call uses Grafana to jump from alert to top error traces. Step-by-step implementation:

1) On alert, query high-error-rate traces grouped by operation. 2) Use trace waterfall to find first failing span. 3) Correlate to deployment events and logs via trace attributes. 4) Capture representative traces and save artifacts for postmortem. What to measure: Error-trace capture rate, first-failure service, deployment timestamps. Tools to use and why: Tempo for traces, CI/CD logs for deploy times, Pager system for escalation. Common pitfalls: Lack of trace retention leading to missing evidence. Validation: Post-incident, verify runbook effectiveness and that trace samples existed. Outcome: Postmortem showed a misconfigured circuit breaker in service C; remediation process updated.

Scenario #4 — Cost vs performance trade-off

Context: Tracing costs rising as traffic quadruples; need to balance cost and observability. Goal: Reduce storage cost while keeping high-fidelity traces for errors. Why Grafana Tempo matters here: Tempo supports sampling and compaction strategies that directly affect cost. Architecture / workflow: Collector applies sampling rules -> Ingester writes sampled chunks -> Compactor reduces storage footprint -> Retention tiering for cold traces. Step-by-step implementation:

1) Measure current trace volume and cost per GB. 2) Implement head-based sampling for low-risk endpoints. 3) Enable tail-based sampling to always keep error and high-latency traces. 4) Introduce retention tiers: hot 7 days, cold 90 days. What to measure: Storage growth, error-trace preservation rate, query latency for cold data. Tools to use and why: Tempo, object storage metrics, billing dashboards. Common pitfalls: Overaggressive sampling dropping rare but critical traces. Validation: Run backfill checks and compare error capture rates before and after sampling. Outcome: Reduced monthly trace storage cost by 60% while retaining >95% of error traces.

Common Mistakes, Anti-patterns, and Troubleshooting

List 15–25 mistakes with: Symptom -> Root cause -> Fix

1) Symptom: Missing spans across services -> Root cause: No context propagation headers -> Fix: Ensure trace context is passed for HTTP and RPC calls. 2) Symptom: Very high storage bills -> Root cause: Full tracing with no sampling -> Fix: Implement head or tail-based sampling and retention tiering. 3) Symptom: Slow trace loads -> Root cause: Cold reads from deep object storage -> Fix: Shorten compactor lag, enable hot cache, or keep recent traces on faster storage. 4) Symptom: Many partial traces -> Root cause: Collector export timeouts or network drops -> Fix: Increase collector batching and retries; check network. 5) Symptom: No traces after deployment -> Root cause: Broken exporter configuration in collector -> Fix: Validate collector config and test synthetic traces. 6) Symptom: Queries error 5xx -> Root cause: Querier resource exhaustion -> Fix: Scale queriers and add rate limiting. 7) Symptom: Alerts noisy during deploys -> Root cause: Alerts tied to raw error counts -> Fix: Use deployment suppression or alert on SLO burn rate. 8) Symptom: Trace attributes missing -> Root cause: Instrumentation not adding necessary tags -> Fix: Standardize attribute enrichment in SDKs. 9) Symptom: Sampling skew -> Root cause: Static head sampling across variable traffic -> Fix: Use adaptive or tail-based sampling for consistent error capture. 10) Symptom: Compactor backlog accumulates -> Root cause: Insufficient compactor instances or CPU limits -> Fix: Scale compactor and tune resource limits. 11) Symptom: Unauthorized writes to object store -> Root cause: Wrong IAM or credentials -> Fix: Rotate keys and enforce least privilege. 12) Symptom: High CPU on ingesters -> Root cause: Large chunk compression overhead -> Fix: Tune chunk size and CPU requests. 13) Symptom: Inconsistent service maps -> Root cause: Fragmented instrumentation patterns -> Fix: Standardize instrumentation and naming conventions. 14) Symptom: Traces have PII -> Root cause: Sensitive data in span attributes -> Fix: Sanitize or drop sensitive attributes before export. 15) Symptom: Slow tail trace capture -> Root cause: Late decision tail sampling not configured properly -> Fix: Configure tail sampling processor and adequate buffer. 16) Symptom: Trace lookups by operation fail -> Root cause: Missing or delayed metadata index -> Fix: Check compactor job status and indexing configuration. 17) Symptom: Alerts missing for SLO breaches -> Root cause: Incorrect SLI computation from traces -> Fix: Verify query logic and input trace completeness. 18) Symptom: Traces duplicated -> Root cause: Multiple collectors exporting same spans -> Fix: Deduplicate at ingestion or remove duplicates in exporter. 19) Symptom: High egress cost when querying -> Root cause: Querier fetching many chunks across cloud regions -> Fix: Co-locate Tempo components with storage or use region-aligned storage. 20) Symptom: Long tail traces hard to parse -> Root cause: Overly fine-grained span creation -> Fix: Merge spans or increase sampling on noisy endpoints. 21) Symptom: CQRS or async workflows not represented -> Root cause: No correlation IDs in async messages -> Fix: Inject trace context into message payloads. 22) Symptom: Test environments pollute production traces -> Root cause: Shared exporters or labels -> Fix: Separate namespaces and labels for envs. 23) Symptom: Query timeouts -> Root cause: Misconfigured timeout on Grafana or querier -> Fix: Increase timeouts or improve query performance. 24) Symptom: Unexpected trace deletion -> Root cause: Retention misconfiguration or lifecycle rules -> Fix: Verify retention policy and object lifecycle rules. 25) Symptom: On-call confusion over trace ownership -> Root cause: No service ownership mapping in traces -> Fix: Add team or owner attributes to spans and dashboards.

Observability pitfalls (subset)

  • Over-reliance on averages for SLOs -> Fix: use p95/p99 from traces.
  • Correlation gaps between logs and traces -> Fix: add correlation ID in logs.
  • Instrumentation blind spots -> Fix: instrument async and background jobs.
  • Misleading sampling artifacts -> Fix: track sampling rates and adjust.
  • No validation in CI for instrumentation -> Fix: add telemetry checks in pipeline.

Best Practices & Operating Model

Cover:

  • Ownership and on-call
  • Runbooks vs playbooks
  • Safe deployments
  • Toil reduction and automation
  • Security basics

Ownership and on-call

  • Assign clear ownership of tracing platform to a platform or observability team.
  • Application teams own instrumentation and SLO definitions.
  • Maintain an on-call rotation for platform health and a separate on-call for application SLOs.

Runbooks vs playbooks

  • Runbooks: Step-by-step, low-complexity recovery procedures for known issues (restart collector, reapply compactor job).
  • Playbooks: Higher-level decision guides covering escalation and postmortem steps.
  • Keep runbooks short and validated during game days.

Safe deployments

  • Canary tracing: Deploy instrumentation changes incrementally and monitor trace volumes.
  • Rollback triggers: If trace ingestion drops or p99 worsens for critical flows, rollback automatically.
  • Use feature flags for instrumentation toggles to reduce blast radius.

Toil reduction and automation

  • Automate scaling triggers for ingesters/queriers based on ingestion and query latency.
  • Automate instrumentation checks in CI to prevent regressions.
  • Automate daily or weekly reports for top error traces and team ownership.

Security basics

  • Avoid sending PII in span attributes; apply client-side sanitization.
  • Encrypt data in transit with TLS and use IAM roles for object store access.
  • Enforce least-privilege access for Tempo components and dashboards.

Weekly/monthly routines

  • Weekly: Review top error traces, alert noise, and sampling effectiveness.
  • Monthly: Evaluate storage growth, adjust retention and sampling settings.
  • Quarterly: Audit access permissions and refresh runbooks.

Postmortem review items related to Tempo

  • Confirm whether traces captured critical evidence.
  • Verify sampling and retention settings were adequate for incident analysis.
  • Add instrumentation to missing code paths discovered during postmortem.

What to automate first

  • Sampling rule enforcement and monitoring.
  • Trace coverage checks in CI.
  • Scaling policies for ingesters and queriers.

Tooling & Integration Map for Grafana Tempo (TABLE REQUIRED)

ID Category What it does Key integrations Notes
I1 Instrumentation SDK Emits spans from apps OpenTelemetry SDKs, language libs Core for trace generation
I2 Collector Aggregates and exports telemetry Tempo, Prometheus, logging Central pipeline point
I3 Object storage Stores trace chunks S3 GCS Azure blob compatible Durable storage for chunks
I4 Grafana Visualization and alerting Tempo data source, Loki, Prometheus User-facing UI
I5 Service mesh Automatic context propagation Istio, Linkerd, Envoy Simplifies propagation
I6 CI/CD Deployment and telemetry checks GitHub Actions, GitLab pipelines Validates instrumentation
I7 Log store Stores logs for correlation Loki, Elasticsearch, Cloud logs Enables trace-to-log linking
I8 Metrics store Stores metrics and SLIs Prometheus, Cortex, Thanos For SLOs and dashboards
I9 Alerting Routing notifications Alertmanager, Pager systems SLO-driven alerting
I10 Security tools Access control and audit SIEM, IAM, Secrets manager Protects trace data
I11 Cost monitoring Tracks Opex related to tracing Billing tools and dashboards Drives retention decisions
I12 Chaos tooling Simulates failures Chaos Mesh, Litmus Validates runbooks

Row Details (only if needed)

  • None

Frequently Asked Questions (FAQs)

H3: What is Grafana Tempo best used for?

Answer: Distributed tracing backend for reconstructing request flows across services and correlating with logs and metrics.

H3: How do I instrument my application for Tempo?

Answer: Use OpenTelemetry SDKs or compatible tracing libraries to emit spans and configure a collector or exporter that sends data to Tempo.

H3: How does Tempo store traces?

Answer: Tempo stores compressed trace chunks in object storage and uses minimal indexes; object storage durability handles retention.

H3: What’s the difference between Tempo and Jaeger?

Answer: Jaeger includes a storage and UI stack; Tempo is index-light and built for cost-efficient storage at scale and integrates with Grafana for UI.

H3: What’s the difference between Tempo and Zipkin?

Answer: Zipkin is a tracing format and lightweight server; Tempo focuses on scalable chunk-based storage and reconstructions.

H3: What’s the difference between Tempo and Prometheus?

Answer: Tempo stores traces and spans; Prometheus stores time-series metrics. They are complementary.

H3: How do I reduce tracing cost?

Answer: Implement head and tail-based sampling, retention tiering, and monitor storage growth; automate sampling rules for high-volume services.

H3: How long should I retain traces?

Answer: Varies per compliance and forensic needs; common patterns are hot 7–30 days and cold 90–365 days depending on business requirements.

H3: How do I ensure error traces are not lost?

Answer: Use tail-based sampling that favors errors and high-latency traces and track error-trace capture rate metrics.

H3: How does Tempo handle large-scale ingestion?

Answer: Scale ingesters, distributors, and provide sufficient object storage throughput; tune collectors for batching and backoff.

H3: How do I correlate traces with logs?

Answer: Ensure logs include trace or correlation IDs, add those IDs to log lines, and link in Grafana via queries.

H3: How do I test tracing in CI?

Answer: Run synthetic requests during deploy stages and assert traces exist for known trace IDs or flows.

H3: How do I debug missing traces?

Answer: Check collector exporter logs, verify context propagation, validate sampling rules, and examine object storage writes.

H3: How do I secure trace data?

Answer: Encrypt in transit, use least privilege for storage access, sanitize sensitive attributes, and restrict dashboard access.

H3: How do I measure Tempo performance?

Answer: Track ingest rate, query latency, compactor lag, and storage growth using Prometheus and Grafana dashboards.

H3: How do I set trace sampling rates?

Answer: Start low for high-volume endpoints and use tail-based sampling to retain errors; iterate based on storage metrics.

H3: How do I avoid PII in traces?

Answer: Sanitize attributes in instrumentation code or use processors in the collector to drop or hash sensitive fields.

H3: How do I choose between managed and self-hosted Tempo?

Answer: Evaluate team capacity, compliance needs, performance SLAs, and cost predictability; managed reduces operational burden.

Conclusion

Summarize and provide a “Next 7 days” plan (5 bullets).

Summary

  • Grafana Tempo is a scalable, cost-conscious distributed tracing backend designed to reconstruct traces and integrate with metrics and logs.
  • Use Tempo where distributed causality matters and combine it with robust instrumentation, sampling, and SLOs.
  • Operational success depends on clear ownership, validated runbooks, and continuous measurement of trace health and cost.

Next 7 days plan

  • Day 1: Inventory critical services and instrument two top user flows with OpenTelemetry.
  • Day 2: Deploy a collector and configure Tempo exporter with basic sampling and test writes.
  • Day 3: Create executive and on-call dashboards for ingest and query latency.
  • Day 4: Define one SLO based on trace-derived latency and configure burn-rate alerts.
  • Day 5–7: Run a load test, validate trace completeness, tune sampling, and document runbooks.

Appendix — Grafana Tempo Keyword Cluster (SEO)

Return 150–250 keywords/phrases grouped as bullet lists only:

  • Primary keywords
  • Related terminology

Primary keywords

  • Grafana Tempo
  • Tempo tracing
  • distributed tracing Tempo
  • Tempo vs Jaeger
  • Tempo tutorial
  • Tempo guide
  • Tempo tracing backend
  • Tempo object storage
  • Tempo compactor
  • Tempo querier
  • Tempo ingestion
  • Tempo setup
  • Tempo architecture
  • Tempo best practices
  • Tempo SLOs
  • Tempo sampling
  • Tempo retention
  • Tempo optimization
  • Tempo troubleshooting
  • Tempo for Kubernetes
  • Tempo for serverless
  • Tempo performance
  • Tempo monitoring
  • Tempo costs
  • Tempo security
  • Tempo deployment
  • Tempo scaler
  • Tempo configuration
  • Tempo dashboards
  • Tempo alerts
  • Tempo integration

Related terminology

  • distributed tracing
  • trace span
  • span attributes
  • trace ID
  • parent ID
  • OpenTelemetry tracing
  • OTel collector
  • head-based sampling
  • tail-based sampling
  • trace compaction
  • object storage tracing
  • S3 tracing storage
  • cold reads
  • hot cache
  • trace chunk
  • ingester component
  • compactor component
  • querier component
  • distributor component
  • service map
  • trace reconstruction
  • trace completeness
  • trace retention policy
  • trace sampling strategy
  • trace enrichment
  • correlation ID tracing
  • trace to log correlation
  • trace to metric correlation
  • CI trace validation
  • tracing in CI/CD
  • tracing runbook
  • tracing playbook
  • tracing alerts
  • trace error budget
  • SLO burn rate tracing
  • trace privacy sanitization
  • trace PII removal
  • trace attribute naming
  • trace instrumentation library
  • trace propagation headers
  • async trace correlation
  • serverless trace cold start
  • mesh tracing Istio
  • mesh tracing Envoy
  • trace export retries
  • trace batching
  • trace compression
  • trace chunk size
  • compactor lag metric
  • query latency metric
  • ingestion error metric
  • trace storage growth
  • trace cost optimization
  • trace retention tiering
  • trace lifecycle management
  • trace query federation
  • trace UI Grafana
  • trace visualize waterfall
  • trace waterfall view
  • trace waterfall latency
  • trace error root cause
  • trace debugging workflow
  • trace incident response
  • trace postmortem evidence
  • trace forensic analysis
  • trace ownership tagging
  • trace team attribution
  • trace telemetry pipeline
  • trace observability pipeline
  • trace monitoring best practices
  • tracing KPIs
  • tracing maturity ladder
  • tracing beginner checklist
  • tracing production checklist
  • tracing preproduction checklist
  • tracing incident checklist
  • tracing load testing
  • tracing chaos testing
  • tracing game day
  • tracing adaptive sampling
  • tracing quota management
  • tracing rate limits
  • tracing exponential backoff
  • tracing retries
  • tracing throughput tuning
  • tracing storage IOPS
  • tracing egress costs
  • tracing regional storage
  • tracing cross region reads
  • tracing ingestion scaling
  • tracing querier scaling
  • tracing ingest rate
  • tracing query rate
  • tracing per service metrics
  • tracing per operation metrics
  • tracing tail latency
  • tracing p95 p99
  • tracing p50 p75
  • tracing percentiles
  • tracing histogram
  • tracing latency distribution
  • tracing error traces preservation
  • tracing sampling bias
  • tracing coverage
  • tracing SDKs language support
  • tracing Java SDK
  • tracing Python SDK
  • tracing Go SDK
  • tracing Node SDK
  • tracing .NET SDK
  • tracing Ruby SDK
  • tracing PHP SDK
  • tracing telemetry export
  • tracing exporter configuration
  • tracing TLS transport
  • tracing IAM permissions
  • tracing object store ACLs
  • tracing secure storage
  • tracing role based access
  • tracing audit trails
  • tracing retention compliance
  • tracing audit retention
  • tracing GDPR considerations
  • tracing PCI considerations
  • tracing SOC compliance
  • tracing managed service
  • tracing SaaS Tempo
  • tracing self hosted Tempo
  • tracing helm chart
  • tracing Kubernetes helm
  • tracing DaemonSet collector
  • tracing gateway collector
  • tracing sidecar collector
  • tracing service mesh integration
  • tracing envoy instrumentation
  • tracing distributed context
  • tracing correlation keys
  • tracing log linking technique
  • tracing query timeout tuning
  • tracing deduplication
  • tracing instrumentation regression
  • tracing CI checks
  • tracing synthetic traces
  • tracing smoke tests
  • tracing health checks
  • tracing alert routing
  • tracing on call practices
  • tracing automation first steps
  • tracing runbook templates
  • tracing playbook templates
  • tracing incident classification
  • tracing root cause analysis
  • tracing remediation workflow
  • tracing rollback triggers
  • tracing canary monitoring
  • tracing deployment observability
  • tracing telemetry governance
  • tracing cost governance
  • tracing observability governance
  • tracing team ownership model
  • tracing telemetry lifecycle
  • tracing keyword cluster
  • tracing seo keywords
  • tracing semantic cluster
  • tracing long tail keywords
  • tracing short tail keywords
Scroll to Top