dynatrace/dynatrace-for-ai· Apache-2.0 内容可用

dt-obs-predictive-analytics

Predictive analytics for Dynatrace — time series forecasting with the timeseries-forecast tool, capacity saturation planning, trend and anomaly detection across hosts, services, and infrastructure.

安装

与 skills.sh 相同的 Command / Prompt 安装方式

name: dt-obs-predictive-analytics description: Predictive analytics for Dynatrace — time series forecasting with the timeseries-forecast tool, capacity saturation planning, trend and anomaly detection across hosts, services, and infrastructure. license: Apache-2.0

Predictive Analytics Skill

Forecast resource saturation, detect trends, analyze anomalies, and characterize signal behavior using DQL and Dynatrace analyzer tools.

Analysis Disciplines

#	Discipline	Use when …
1	Forecast and Prediction	Predicting future metric values for capacity planning, cost estimation, or proactive alerting
2	Detecting Changes	A metric shifted — find when the character of the signal changed, regardless of whether it crossed a limit
3	Detecting Violations	A metric is currently out of bounds — find entities that exceed or fall below an acceptable range
4	Timeseries Characteristics	Characterizing a signal's seasonality, noise level, and trend before further analysis

Choosing the Right Detection Tool

The single most important decision: are you asking "did this metric change?" or "is this metric currently wrong?"

Question	Tool	Why
"Did this metric change in the last N hours?"	`timeseries-novelty-detection`	Detects when the signal's character changed (spike, step, trend onset, variability shift) without requiring a known acceptable limit
"Which services spiked or dropped recently?"	`timeseries-novelty-detection` with `SPIKE` / `CHANGE_IN_VALUES`	Finds the specific entities and timestamps where change occurred; returns empty for stable signals
"When did CPU start trending up?"	`timeseries-novelty-detection` with `TREND_IN_VALUES`	Pinpoints the onset of a directional shift
"Which hosts are currently above 90% CPU?"	`static-threshold-analyzer`	Known fixed limit — fire alerts when exceeded
"Which services are currently above their usual load?"	`adaptive-anomaly-detector`	Learns the normal distribution from the data and flags sustained threshold violations
"Which services are high right now vs. their weekly pattern?"	`seasonal-baseline-anomaly-detector`	Accounts for time-of-day/day-of-week patterns before deciding what is anomalous

Decision rule in plain language

Use timeseries-novelty-detection when the question contains "changed", "shifted", "spiked", "dropped", "started", "when did", or "did anything unusual happen". The tool answers whether a change occurred and when. It requires no predefined threshold.
Use an anomaly detector (adaptive, seasonal, or static) when the question is about ongoing or current state relative to an expected range: "which are highest", "who is violating", "what is above X". These tools count violation samples inside a sliding window — they confirm how long something has been bad, not whether the signal changed.

Pitfall: Running adaptive-anomaly-detector on a broad fleet to answer "which service changed load?" typically flags every service that has any variation, producing low-signal results. Use timeseries-novelty-detection first to identify entities where the load character genuinely shifted, then use the anomaly detectors to measure the severity of those specific signals.

When to Use This Skill

Capacity: "Which hosts will hit 90% CPU in the next 30 days?"
Forecast: "Forecast service request volume for the next 7 days"
Trend: "Is memory usage growing across our Kubernetes nodes?"
Anomaly: "Which services have unusual error rates right now?"
Baseline: "How does today's traffic compare to last week?"
Signal profile: "Is this metric seasonal or trending before I set up alerting?"

Important Constraints

Dynatrace Forecast Analyzer supports univariate forecasting only — predicting one metric based on its own historical values. Multivariate forecasting (using multiple metrics as inputs) requires external tools (Python, R, Azure AutoML).

Tooling Rule: Run analyses using Dynatrace tools: timeseries-forecast, adaptive-anomaly-detector, seasonal-baseline-anomaly-detector, static-threshold-analyzer, and timeseries-novelty-detection. Use execute-dql for DQL queries.

Result Analysis Rule: Always analyse and summarise results directly from the raw tool output. Derive all numbers, trends, and conclusions inline.

Result Presentation Format

Always present forecast results as a structured table:

Column	Content
Rank	🥇 🥈 🥉 ordered by urgency or magnitude
Signal / Entity	Metric name and entity or dimension
Last Actual	Most recent non-null value from the historical series
Forecast	Point forecast at the end of the horizon
Range	Lower – Upper confidence band at the same horizon point
Trend	% change from Last Actual to Forecast: 🔴 >+20% / 🟠 +5–20% / 🟢 ±5% stable / 🔵 −5–20% declining / ⚫ <−20% sharp drop
Action	✅ No action / ⚠️ Monitor / 🔴 Act now

Always follow the table with a Key Findings section (3–5 bullet points, ranked by priority).

Core DQL Techniques

DQL has no native forecast function. For forward-looking forecasts, use timeseries-forecast (see references/forecasting-analyzer.md).

Key DQL Rules

timeseries returns arrays — one value per time slot per entity
arrayLast(arr) = most recent value; arrayFirst(arr) = oldest
Growth = (arrayLast - arrayFirst) / number_of_intervals
Always filter isNotNull(field) before sorting to avoid null ordering issues
Use toLong() when dividing Long fields to avoid type errors
Use dt.smartscape.* not deprecated dt.entity.* in DQL display fields; use dt.smartscape.* in by:{} grouping clauses for entity-level queries

Standard Query Patterns

Moving Average Trend

timeseries cpu = avg(dt.host.cpu.usage), from: now()-24h, interval: 1h, by: {dt.smartscape.host}
| fieldsAdd moving_avg = arrayMovingAvg(cpu, 4)
| fieldsAdd current = arrayLast(cpu)
| fieldsAdd trend = arrayLast(cpu) - arrayFirst(cpu)
| filter isNotNull(current)
| sort trend desc
| limit 20
| fields dt.smartscape.host, current, trend, moving_avg

Saturation Risk Classification

timeseries cpu = avg(dt.host.cpu.usage), from: now()-7d, interval: 1h, by: {dt.smartscape.host}
| fieldsAdd p95 = arrayPercentile(cpu, 95)
| fieldsAdd saturation_risk = if(p95 > 85, "HIGH", else: if(p95 > 70, "MEDIUM", else: "LOW"))
| filter isNotNull(p95)
| sort p95 desc
| fields dt.smartscape.host, p95, saturation_risk

Days to Saturation Forecast

timeseries cpu = avg(dt.host.cpu.usage), from: now()-30d, interval: 1d, by: {dt.smartscape.host}
| fieldsAdd current = arrayLast(cpu)
| fieldsAdd daily_growth = (arrayLast(cpu) - arrayFirst(cpu)) / 30
| filter isNotNull(current)
| fieldsAdd days_to_saturation = if(daily_growth > 0, toLong((90 - current) / daily_growth), else: 9999)
| sort days_to_saturation asc
| limit 20
| fields dt.smartscape.host, current, daily_growth, days_to_saturation

Anomaly Scoring

timeseries cpu = avg(dt.host.cpu.usage), from: now()-24h, interval: 1h, by: {dt.smartscape.host}
| fieldsAdd baseline_avg = arrayAvg(cpu)
| fieldsAdd current = arrayLast(cpu)
| fieldsAdd anomaly_score = if(isNotNull(current) and isNotNull(baseline_avg), abs(current - baseline_avg), else: 0)
| sort anomaly_score desc
| limit 20
| fields dt.smartscape.host, current, baseline_avg, anomaly_score

Metric Discovery

Before forecasting, discover available metrics by keyword:

metrics from: now() - 1h
| filter contains(metric.key, "cpu")
| summarize count(), by: {metric.key}
| sort `count()` desc

Reference Guides

references/forecasting-analyzer.md — timeseries-forecast tool: data requirements, parameter reference, interval selection, horizon limits, common pitfalls
references/capacity-forecasting.md — CPU/memory/disk/K8s saturation forecasts; multi-resource risk scoring; days-to-saturation DQL patterns
references/anomaly-scoring.md — adaptive-anomaly-detector, seasonal-baseline-anomaly-detector, static-threshold-analyzer; DQL deviation scoring
references/novelty-detection.md — timeseries-novelty-detection tool: spike, drop, step change, trend onset, and variability change detection; all novelty types; parameter reference; worked examples
references/trend-detection.md — timeseries-novelty-detection for trend onset and change points; week-over-week joins; growth rate and acceleration detection

Related Skills

dt-dql-essentials — DQL syntax, timeseries command rules, array function reference
dt-obs-hosts — Host and process metrics catalog
dt-obs-services — Service RED metrics for service-level trend analysis
dt-obs-problems — Davis AI problem history for anomaly correlation

附带文件

references/anomaly-scoring.md

# Anomaly Scoring and Detection

Detect anomalies using Dynatrace analyzer tools. Use the DQL patterns
below for transparent, tunable scoring when you need explicit deviation
numbers alongside the analyzer output.

---

## Dynatrace Anomaly Detection Tools

> **When to use this reference vs. novelty detection**
>
> The three anomaly detector tools (`adaptive`, `seasonal`, `static`) answer **"is this metric currently violating an expected range?"** They operate on a sliding window of recent samples and fire when enough consecutive samples breach a learned or fixed threshold. Use them to rank entities by ongoing severity or to confirm a sustained problem.
>
> Use `timeseries-novelty-detection` (see `references/novelty-detection.md`) when the question is **"did this metric change?"** — no threshold required, answers *whether* the signal's character shifted and *when*. Running an anomaly detector fleet-wide to find "what changed" typically flags every service with any variation; novelty detection is the correct tool for that question.

| Tool | Answers | Best For |
|------|---------|---------|
| `adaptive-anomaly-detector` | "Is it currently above/below normal?" | General-purpose; learns threshold from signal distribution |
| `seasonal-baseline-anomaly-detector` | "Is it above/below normal *for this time of day/week*?" | Metrics with daily/weekly/variable seasonality |
| `static-threshold-analyzer` | "Is it above/below a fixed limit?" | Known, fixed upper/lower limits (e.g., CPU > 90%) |
| `timeseries-novelty-detection` | "Did the signal's behavior change?" | Step changes, spikes, trend onset — no threshold needed |

All tools return a `raisedAlerts` array per entity; an empty `output` array
means no anomalies were detected in the analysis window.

### Adaptive Anomaly Detection

Learns a dynamic threshold from the signal's own distribution.
Good default for most host and service metrics.

**Tool**: `adaptive-anomaly-detector`
- `timeSeriesData`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:1m | limit 20`
- `numberOfSignalFluctuations`: `1.0` — multiplier for threshold width; increase to reduce sensitivity
- `alertCondition`: `ABOVE`
- `violatingSamples`: `3` — consecutive violations required before alerting
- `slidingWindow`: `5`
- `generalParameters.timeframe.startTime`: `now-24h`

### Seasonal Baseline Anomaly Detection

Accounts for daily/weekly patterns. `tolerance` controls sensitivity —
lower value = more alerts (valid range 0.1–10.0; default 4.0).

**Tool**: `seasonal-baseline-anomaly-detector`
- `timeSeriesData`: `timeseries sum(dt.service.request.count), by:{dt.entity.service}, interval:5m | limit 10`
- `tolerance`: `3.0`
- `violatingSamples`: `3`
- `slidingWindow`: `5`
- `alertCondition`: `OUTSIDE`
- `generalParameters.timeframe.startTime`: `now-7d`

### Static Threshold Detection

Use when the acceptable limit is known and stable (e.g., CPU > 90%).
`threshold` must be in the base unit of the queried metric (response time
metrics use nanoseconds — set 5 seconds as `5000000000`, not `5000`).

**Tool**: `static-threshold-analyzer`
- `timeSeriesData`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:1m | limit 20`
- `threshold`: `90`
- `alertCondition`: `ABOVE`
- `violatingSamples`: `3`
- `slidingWindow`: `5`
- `generalParameters.timeframe.startTime`: `now-24h`

### Novelty Detection

Detects previously unseen patterns, including step changes, trend onsets,
and isolated spikes. Use `analysisNoveltyType` to focus on specific event types.

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries sum(dt.service.request.failure_count), by:{dt.entity.service}, interval:5m | limit 10`
- `detectionMode`: `ALL`
- `minNoveltyScore`: `0.45` (0.0–1.0; higher = fewer but more confident detections)
- `significanceLevel`: `HIGH`
- `analysisNoveltyType`: `["CHANGE_IN_VALUES", "SPIKE", "TREND_IN_VALUES"]`
- `generalParameters.timeframe.startTime`: `now-7d`

Available novelty types: `CHANGE_IN_VALUES`, `CHANGE_IN_VARIABILITY`,
`CHANGE_IN_MISSING_VALUES`, `GAP_WITH_MISSING_VALUES`, `TREND_IN_VALUES`, `SPIKE`.

---

## Anomaly Types

| Type | Description | Detection Approach |
|------|-------------|-------------------|
| **Point** | Single isolated spike | `adaptive-anomaly-detector` + DQL p95 |
| **Contextual** | Anomalous only in a specific context (e.g., 3 AM spike) | `seasonal-baseline-anomaly-detector` |
| **Collective** | Sustained group deviation | `adaptive-anomaly-detector` with wider `slidingWindow` |
| **Step change** | Abrupt permanent baseline shift | `timeseries-novelty-detection` with `CHANGE_IN_VALUES` |
| **Novel pattern** | Previously unseen shape | `timeseries-novelty-detection` with `SPIKE` or `TREND_IN_VALUES` |

---

## Result Presentation

| Column | Content |
|--------|---------|
| Rank | 🥇 🥈 🥉 by deviation magnitude |
| Signal / Entity | Metric name and entity |
| Observed | Actual value at anomaly point |
| Expected | Baseline or forecast band |
| Deviation | % or absolute deviation |
| Type | Point / Contextual / Collective / Step / Novel |
| Action | ✅ Monitor / ⚠️ Investigate / 🔴 Escalate |

---

## Best Practices

- Compare anomaly candidates against the same time window in the prior
  day/week before escalating
- Consider seasonality — a spike at Monday 9 AM may be expected
- Correlate Davis AI findings with DQL scores to prioritize what matters
- `dt.service.request.response_time` is in nanoseconds — divide by
  `1000000` for ms before scoring
- For `static-threshold-analyzer`, always set `threshold` in the metric's
  base unit (nanoseconds for response time, percent for CPU)
- `violatingSamples` must not exceed `slidingWindow`

references/capacity-forecasting.md

# Capacity Forecasting

Forecast resource saturation for hosts, Kubernetes workloads, and billing
metrics. Use `timeseries-forecast` for forward-looking projections; use
the DQL patterns below for immediate risk classification.

---

## Dynatrace Forecast Analyzer

Use `timeseries-forecast` for all capacity forecasts. See
`references/forecasting-analyzer.md` for the full parameter reference,
feasibility decision tree, and horizon sizing rules.

**Key rule**: For 30-day forecasts, use `interval:1d` and `forecastHorizon:30`.
`forecastHorizon` is in steps (intervals), not absolute days.

### Host CPU — 30-Day Saturation Forecast

Training: 120 days. Interval: `1d`. Horizon: 30 daily steps.

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:1d | limit 500`
- `forecastHorizon`: `30`
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-60d`

### Host Memory — 30-Day Saturation Forecast

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.host.memory.usage), by:{dt.entity.host}, interval:1d | limit 500`
- `forecastHorizon`: `30`
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-120d`

### Kubernetes Container CPU — Namespace Forecast

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.kubernetes.container.cpu_usage), by:{k8s.namespace.name, k8s.workload.name}, interval:1d | limit 500`
- `forecastHorizon`: `30`
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-60d`

### Kubernetes PVC Storage Saturation

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.kubernetes.persistentvolumeclaim.used), by:{k8s.namespace.name}, interval:1d | limit 500`
- `forecastHorizon`: `30`
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-60d`

---

## Result Presentation

| Column | Content |
|--------|---------|
| Rank | 🥇 🥈 🥉 by days to saturation (ascending) |
| Host / Workload | Entity identifier |
| Current (%) | Latest measured utilization |
| Growth/Day | Daily growth rate in percentage points |
| Days to 90% | Estimated days until saturation at current growth |
| Action | ✅ Safe / ⚠️ Plan ahead / 🔴 Act now |

---

## Best Practices

- Use at least 60 days of history for 30-day forecasts (2:1 rule)
- Use `interval:1d` for 30-day forecasts; `interval:1h` for 7-day
- Saturation threshold is typically 90%; adjust per SLA
- Always `filter isNotNull(field)` before `sort`
- Use `if(daily_growth > 0, ..., else: 9999)` — negative growth = no saturation risk
- Entities with data gaps in the most recent third of the training window will
  fail forecasting; filter them out or shorten the training window

references/forecasting-analyzer.md

# Dynatrace Forecast Analyzer

The `timeseries-forecast` tool generates time series forecasts for any numeric
metric queryable from Grail. It uses Davis AI statistical models to produce
point forecasts and confidence bands.

**⚠️ Univariate only**: Dynatrace Forecast Analyzer currently supports only
univariate forecasting. See Section 1 for details and multivariate alternatives.

**🚫 Result Analysis Rule**: After running a forecast or DQL query, always
analyse and summarise the results directly from the raw tool output.

**📊 Result Presentation Rule**: Present forecast results as a structured
Markdown table:

| Column | Content |
|--------|---------|
| Rank | 🥇 🥈 🥉 4️⃣ … ordered by urgency or magnitude of change |
| Signal / Entity | Dimension name (entity, product, service, etc.) |
| Last Actual | Most recent non-null value from the historical series |
| Forecast D+N | Point forecast at the end of the horizon |
| Range | Lower – Upper confidence band at the same horizon point |
| Trend | % change from Last Actual to Forecast D+N: 🔴 >+20%, 🟠 +5–20%, 🟢 ±5% stable, 🔵 −5–20% declining, ⚫ <−20% sharp drop |
| Action | ✅ No action / ⚠️ Monitor / 🔴 Act now |

After the table, add a **Key Findings** section (3–5 bullet points) ranked by priority.

---

## Tool Reference: `timeseries-forecast`

### Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `query` | Yes | DQL `timeseries` query providing the historical series to forecast. Always include `\| limit 50` (or lower). Specify `interval:` explicitly for predictable results. |
| `forecastHorizon` | No | Steps to forecast ahead. **Max 600.** Default 100. Steps × interval = forecast time span. |
| `forecastOffset` | No | Steps to withhold from training end for validation (0–10). Default 1. |
| `generalParameters.timeframe.startTime` | No | Training window start, e.g. `now-120d`. Default: last 2 hours. |
| `generalParameters.timeframe.endTime` | No | Training window end. Default: `now`. |

### Output Fields

The result timeseries contains these fields per record:
- **`<metric_expression>`** — historical values (nulls in the forecast period)
- **`dt.davis.forecast:point`** — point forecast (nulls in the historical period)
- **`dt.davis.forecast:lower`** — lower confidence bound
- **`dt.davis.forecast:upper`** — upper confidence bound

The forecast begins at the last non-null historical value. Read the point
forecast array from the first non-null position onward.

### Common Failure: Missing Values in Recent History

The analyzer requires **at least 14 non-null values in the last third** of the
training window. If the series has too many gaps near the end, the analysis
status will be `FAILED` with a warning:

> "Too many values are missing from the most recent history."

Fix: filter entities with sufficient continuous data, or shorten the training
window to exclude inactive periods.

### Interval and Horizon Sizing

`forecastHorizon` is counted in **steps** (intervals), not in absolute time:
`forecast time = forecastHorizon × interval`.

| Forecast Target | Recommended `interval` | `forecastHorizon` | Training Window |
|-----------------|------------------------|-------------------|-----------------|
| 24 hours ahead  | `interval:1h`          | `24`              | `now-2d`        |
| 7 days ahead    | `interval:1h`          | `168`             | `now-14d`       |
| 30 days ahead   | `interval:1d`          | `30`              | `now-60d`       |
| 90 days ahead   | `interval:1d`          | `90`              | `now-180d`      |

Do not omit `interval:` from the query — without it, the tool auto-selects an
interval that may differ from what you expect, making the horizon ambiguous.

---

## 1. Forecast and Prediction

### Core Principles

#### Univariate vs. Multivariate Forecasting

**Univariate Forecasting**: Predicts a single metric based on its own historical values.
- **Best for**: Stable patterns with strong autocorrelation (e.g., daily traffic patterns)
- **Status with Dynatrace**: ✅ **Supported** via `timeseries-forecast`

**Multivariate Forecasting**: Predicts a target metric using multiple related metrics as features.
- **Best for**: Complex systems where multiple factors drive the target metric
- **Status with Dynatrace**: ⏳ **Not Supported** by Dynatrace Forecast Analyzer. Use external tools (Python, R, Azure AutoML) or approximate with multiple univariate forecasts.

#### Statistical Foundation

Time series forecasting works best when:
- **Stationarity**: Statistical properties (mean, variance) remain roughly constant
- **Seasonality**: Regular, repeating patterns exist (hourly, daily, weekly)
- **Trend**: Long-term direction (upward, downward, or flat)
- **Autocorrelation**: Current values relate to past values

### Critical Data Requirements

#### Minimum Historical Data

- **Hard minimum**: 20 non-null data points
- **Recency requirement**: At least 14 non-null values must fall in the last third of the training window
- **Practical rule**: Ensure continuous data collection with no more than 5% gaps

#### The 2:1 Rule (Data Science Best Practice)

Training history must be at least **2× the forecast horizon**:

| Forecast Horizon | Minimum Training | Example |
|-----------------|------------------|---------|
| 1 day           | 2 days           | `forecastHorizon:24` (1h interval), `startTime:now-2d` |
| 1 week          | 2 weeks          | `forecastHorizon:168` (1h interval), `startTime:now-14d` |
| 30 days         | 60 days          | `forecastHorizon:30` (1d interval), `startTime:now-60d` |

#### Data Quality Rules

1. **No more than 5% missing** data points within the training window
2. **Sufficient variance**: CV (std dev / mean) > 5%; flat metrics cannot be forecast
3. **Complete seasonal cycles**: Include ≥ 2 full cycles (e.g., 2 weeks for weekly patterns)
4. **Clean outliers**: Incidents and one-off spikes distort models; consider excluding them

### Metric Discovery

Before forecasting, use `execute-dql` to confirm which metrics are available:

```dql
fetch metric.series
| filter contains(metric.key, "cpu")
| dedup metric.key
| limit 100
```

Change `"cpu"` to any keyword (e.g., `"request"`, `"memory"`, `"kubernetes"`) to
explore other domains.

### Query Patterns for Forecast Input

Always pair a DQL query with `timeseries-forecast`. The query must return a
`timeseries` result; `limit` controls how many entities are analyzed.

**Host CPU (univariate, by host)**:
```dql
timeseries avg(dt.host.cpu.usage), by:{dt.smartscape.host}, interval:1d | limit 50
```

**Service request volume (univariate, by service)**:
```dql
timeseries sum(dt.service.request.count), by:{dt.entity.service}, interval:1h | limit 50
```

**Single entity forecast** (filter first, then forecast):
```dql
timeseries sum(dt.service.request.count), by:{dt.entity.service}, interval:1h
| filter dt.entity.service == "SERVICE-XXXXXXXX"
```

### Do's and Don'ts

#### ✅ DO

- Specify `interval:` explicitly so the horizon count is predictable
- Use at least 3× training history relative to the forecast horizon
- Include ≥ 2 complete seasonal cycles in training data
- Retrain regularly — patterns change after deployments and events
- Set `forecastOffset: 1` to validate the last point against forecast

#### ❌ DON'T

- Forecast with fewer than 20 data points
- Omit `interval:` from the query (leads to ambiguous horizon sizing)
- Set `forecastHorizon` > 600 (hard limit)
- Forecast beyond 2–3× the training window (uncertainty explodes)
- Use a metric with CV < 5% (flat signals produce meaningless forecasts)
- Mix pre/post-deployment data in the training window

---

## 2. Top Use Cases

### Capacity Planning (30 days)

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:1d | limit 500`
- `forecastHorizon`: `30` (30 daily steps = 30 days)
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-60d`

Repeat for memory (`dt.host.memory.usage`) and disk (`dt.host.disk.used.percent`).
See `references/capacity-forecasting.md` for multi-resource patterns.

### Traffic Forecasting (7 days)

**Tool**: `timeseries-forecast`
- `query`: `timeseries sum(dt.service.request.count), by:{dt.entity.service}, interval:1h | limit 500`
- `forecastHorizon`: `168` (168 hourly steps = 7 days)
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-14d`

### SLA / Response Time Forecast (24 hours)

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.service.request.response_time), by:{dt.entity.service}, interval:1h | limit 500`
- `forecastHorizon`: `24` (24 hourly steps = 24 hours)
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-7d`

Note: `dt.service.request.response_time` is in microseconds — divide by 1 000 for milliseconds in DQL before displaying.

### Anomaly Baseline Forecasting

Generate a dynamic baseline by running a short-horizon forecast, then alert
when actuals exceed `dt.davis.forecast:upper`:

**Tool**: `timeseries-forecast`
- `query`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:1h | limit 500`
- `forecastHorizon`: `24`
- `forecastOffset`: `1`
- `generalParameters.timeframe.startTime`: `now-2d`

Retrain daily. Alert when `actual > dt.davis.forecast:upper`.

---

## 3. Feasibility Decision Tree

```
Q: Should we forecast this metric?
├─ Do we have ≥ 20 historical data points?
│  ├─ No  → Wait and collect more data
│  └─ Yes → Continue
├─ Does the last third of the window have ≥ 14 non-null values?
│  ├─ No  → Shorten the training window or filter to active entities
│  └─ Yes → Continue
├─ Does the metric have meaningful variance? (CV > 5%)
│  ├─ No  → Skip forecasting; use static thresholds instead
│  └─ Yes → Continue
├─ Is training history ≥ 2× the forecast horizon?
│  ├─ No  → Reduce forecastHorizon or extend startTime
│  └─ Yes → Proceed
└─ Single or multiple metrics?
   ├─ Single → Univariate forecast with timeseries-forecast ✅
   └─ Multiple → External tools (multivariate not supported) ⏳
```

---

## 4. Monitoring Forecast Accuracy

Track these metrics post-deployment:

| Metric | Calculation | Target |
|--------|-----------|--------|
| **MAPE** | `avg(\|actual − forecast\| / actual)` | < 15% |
| **Interval Coverage** | `% of actuals within forecast bounds` | 85–95% |
| **Forecast Bias** | `avg(forecast − actual)` | Near 0 |

Retrain after significant business events (launches, incidents, major config changes).

---

## 5. Common Pitfalls

| Pitfall | Why It Fails | Fix |
|---------|-------------|-----|
| Too many nulls in recent history | Analyzer rejects the series | Filter entities with continuous data or shorten window |
| Omitting `interval:` from query | Horizon count becomes ambiguous | Always specify `interval:` explicitly |
| `forecastHorizon` > 600 | Hard tool limit | Use coarser interval (e.g., `1d` instead of `1h`) |
| Training < 2× horizon | Uncertainty too high | Extend `startTime` |
| Flat metric (CV < 5%) | No pattern to model | Use rule-based alerts instead |
| Pre/post-deployment mix | Model learns old patterns | Set `startTime` after the change |
| Ignoring seasonality | Model misses weekly cycles | Include ≥ 2 weeks of history |

references/novelty-detection.md

# Novelty Detection

Use `timeseries-novelty-detection` to check whether a time series showed a
significant change in the **character** of its behavior during a given period.
Unlike threshold-based detectors, novelty detection does not require a known
acceptable limit — it learns what "normal" looks like and flags anything that
is structurally new: spikes, drops, step changes, shifts in variability, or
the onset of a sustained trend.

---

## When to Use

| Scenario | Example question |
|----------|-----------------|
| **Spike or drop check** | "Did this metric show any sudden spikes in the last 24 h?" |
| **Step change / baseline shift** | "Did CPU usage permanently jump after last night's deployment?" |
| **Trend onset** | "When did memory start trending upward?" |
| **Variability change** | "Is request latency becoming more erratic than before?" |
| **General novel behavior** | "Did anything unusual happen to this signal in the last 7 days?" |

Use this tool as the **first check** when you suspect something changed but
you do not yet know what kind of change it was. Run it with
`analysisNoveltyType: ["SPIKE", "CHANGE_IN_VALUES", "TREND_IN_VALUES"]` and
let the results tell you the category before you narrow in.

**Do not use anomaly detectors (`adaptive-anomaly-detector`, `seasonal-baseline-anomaly-detector`,
`static-threshold-analyzer`) to answer "did this metric change?" questions.**
Those tools count how long a metric stays outside a learned or fixed threshold — they confirm
*ongoing severity*, not the presence of a change. On a broad fleet query they will flag every
service that has *any* variation, producing low-signal results. Novelty detection is the correct
first tool whenever the question contains "changed", "shifted", "spiked", "dropped", "started",
"when did", or "did anything unusual happen".

---

## Tool Parameters

**Tool**: `timeseries-novelty-detection`

| Parameter | Type | Description |
|-----------|------|-------------|
| `timeSeriesData` | DQL string | A `timeseries` query that returns the signal to analyze. Must use `dt.entity.*` fields in `by:{}` grouping (not `dt.smartscape.*`). Add `\| limit N` to control how many entities are analyzed. |
| `detectionMode` | enum | Direction filter: `ALL` (both), `INCREASE` (rises only), `DECREASE` (drops only) |
| `analysisNoveltyType` | array | Which novelty types to look for — see table below |
| `minNoveltyScore` | float 0.0–1.0 | Confidence threshold; `0.45` is a reasonable default. Raise to `0.6–0.8` to reduce false positives. |
| `significanceLevel` | enum | `LOW`, `MEDIUM`, `HIGH` — filters by statistical significance; prefer `HIGH` for production alerting |
| `filterSpikes` | bool | When `true`, transient spikes are removed before trend/change-point analysis. Set `true` when looking for structural changes, `false` when spikes are themselves the signal of interest. |
| `generalParameters.timeframe.startTime` | string | Analysis window start, e.g. `now-7d`, `now-24h` |

### Novelty Types

| `analysisNoveltyType` value | What it detects |
|-----------------------------|----------------|
| `SPIKE` | Short-lived spike (value shoots up then returns) |
| `CHANGE_IN_VALUES` | Abrupt, permanent shift in the mean — step change or baseline jump |
| `TREND_IN_VALUES` | Sustained directional movement starting from a specific point |
| `CHANGE_IN_VARIABILITY` | The signal becomes significantly more or less noisy |
| `CHANGE_IN_MISSING_VALUES` | The rate of null/missing data points changes |
| `GAP_WITH_MISSING_VALUES` | A contiguous gap of missing data appears |

Combine multiple types in one call: `["SPIKE", "CHANGE_IN_VALUES", "TREND_IN_VALUES"]`.

---

## Worked Examples

### Check for any novel behavior (general anomaly sweep)

Use as a first-pass detector across a set of entities when you suspect
something changed but do not know the shape:

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:5m | limit 20`
- `detectionMode`: `ALL`
- `analysisNoveltyType`: `["SPIKE", "CHANGE_IN_VALUES", "TREND_IN_VALUES", "CHANGE_IN_VARIABILITY"]`
- `minNoveltyScore`: `0.45`
- `significanceLevel`: `HIGH`
- `generalParameters.timeframe.startTime`: `now-24h`

### Spike and drop detection (isolated events)

Check whether a service's error rate showed any sudden spikes or drops in a
recent incident window:

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries sum(dt.service.request.failure_count), by:{dt.entity.service}, interval:1m | limit 10`
- `detectionMode`: `ALL`
- `analysisNoveltyType`: `["SPIKE"]`
- `minNoveltyScore`: `0.5`
- `significanceLevel`: `MEDIUM`
- `filterSpikes`: `false`
- `generalParameters.timeframe.startTime`: `now-6h`

### Step change after a deployment

Confirm whether a deployment caused a permanent baseline shift (not just a
transient spike):

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries avg(dt.service.request.response_time), by:{dt.entity.service}, interval:5m | limit 10`
- `detectionMode`: `INCREASE`
- `analysisNoveltyType`: `["CHANGE_IN_VALUES"]`
- `minNoveltyScore`: `0.55`
- `significanceLevel`: `HIGH`
- `filterSpikes`: `true`
- `generalParameters.timeframe.startTime`: `now-48h`

### Trend onset detection

Identify when a gradual upward or downward trend began:

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries avg(dt.host.memory.usage.percent), by:{dt.entity.host}, interval:1h | limit 15`
- `detectionMode`: `INCREASE`
- `analysisNoveltyType`: `["TREND_IN_VALUES"]`
- `minNoveltyScore`: `0.45`
- `significanceLevel`: `HIGH`
- `filterSpikes`: `true`
- `generalParameters.timeframe.startTime`: `now-30d`

### Variability change (signal becoming erratic)

Detect when a metric that was previously stable starts fluctuating:

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries avg(dt.service.request.response_time), by:{dt.entity.service}, interval:5m | limit 10`
- `detectionMode`: `ALL`
- `analysisNoveltyType`: `["CHANGE_IN_VARIABILITY"]`
- `minNoveltyScore`: `0.45`
- `significanceLevel`: `MEDIUM`
- `generalParameters.timeframe.startTime`: `now-14d`

---

## Interpreting Results

The tool returns a `raisedAlerts` array per entity. Each alert includes:
- The **novelty type** that was detected
- A **novelty score** (0.0–1.0) — higher means more confident
- The **time range** where the novelty was detected

An empty `output` array (or no `raisedAlerts` entries) means the signal
showed no significant novel behavior in the analysis window.

| Novelty Score | Interpretation |
|--------------|---------------|
| 0.0–0.44 | Below threshold — not reported (filtered by `minNoveltyScore`) |
| 0.45–0.59 | Probable novelty — worth investigating |
| 0.60–0.79 | Likely novelty — act or escalate |
| 0.80–1.0 | High-confidence novelty — treat as confirmed |

---

## Result Presentation

| Column | Content |
|--------|---------|
| Rank | 🥇 🥈 🥉 by novelty score descending |
| Entity | Host, service, or other entity |
| Novelty Type | SPIKE / CHANGE_IN_VALUES / TREND_IN_VALUES / CHANGE_IN_VARIABILITY |
| Score | 0.0–1.0 confidence |
| Detected Window | Start–end of the novel period |
| Action | ✅ No action / ⚠️ Investigate / 🔴 Escalate |

---

## Best Practices

- Set `filterSpikes: true` when looking for structural changes (step changes,
  trends) — spikes inflate variance and can obscure change-point detection
- Set `filterSpikes: false` when spikes *are* the signal of interest
- Use `INCREASE` / `DECREASE` mode to reduce noise when the anomaly direction
  is known from context (e.g., "CPU went up after deploy")
- Combine with `adaptive-anomaly-detector` or `seasonal-baseline-anomaly-detector`
  for cross-validation: novelty detection finds *when* the character changed;
  threshold detectors confirm *how much* it exceeds acceptable limits
- `dt.service.request.response_time` is in nanoseconds — results are returned
  in that unit; divide by `1000000` when presenting ms values
- Keep `| limit N` in the DQL query low (10–20) for faster results; increase
  only when a full fleet sweep is required

references/trend-detection.md

# Trend Detection

Detect growth trends, week-over-week changes, and acceleration patterns in
service and infrastructure metrics. Use `timeseries-novelty-detection` for
structural break and trend onset detection; use the DQL patterns for growth
rate analysis and week-over-week comparisons.

---

## Dynatrace Trend Analyzers

### Change Point Detection

Detects when a metric's baseline permanently shifted (e.g., after a deployment).
Use this before declaring a trend to confirm it is a true structural change
vs. noise.

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries sum(dt.service.request.count), by:{dt.entity.service}, interval:1h | limit 10`
- `detectionMode`: `ALL`
- `analysisNoveltyType`: `["CHANGE_IN_VALUES", "CHANGE_IN_VARIABILITY"]`
- `minNoveltyScore`: `0.45`
- `significanceLevel`: `HIGH`
- `generalParameters.timeframe.startTime`: `now-30d`

For host CPU upward step changes:

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries avg(dt.host.cpu.usage), by:{dt.entity.host}, interval:1h | limit 10`
- `detectionMode`: `INCREASE`
- `analysisNoveltyType`: `["CHANGE_IN_VALUES"]`
- `minNoveltyScore`: `0.45`
- `significanceLevel`: `MEDIUM`
- `generalParameters.timeframe.startTime`: `now-30d`

### Trend Onset Detection

Detects when a metric starts moving in a sustained new direction (growing
or declining trend beginning):

**Tool**: `timeseries-novelty-detection`
- `timeSeriesData`: `timeseries avg(dt.service.request.response_time), by:{dt.entity.service}, interval:5m | limit 10`
- `detectionMode`: `ALL`
- `analysisNoveltyType`: `["TREND_IN_VALUES"]`
- `minNoveltyScore`: `0.45`
- `significanceLevel`: `HIGH`
- `filterSpikes`: `true`
- `generalParameters.timeframe.startTime`: `now-14d`

---

## DQL Trend Detection Patterns

Use these with `execute-dql` for quantified growth rates.

### Week-over-Week Request Volume Comparison

```dql
timeseries this_week = sum(dt.service.request.count),
  from: now()-7d, interval: 1h, by: {dt.smartscape.service}
| join [
    timeseries last_week = sum(dt.service.request.count),
      from: now()-14d, to: now()-7d, interval: 1h,
      by: {dt.smartscape.service}
  ], on: {left[dt.smartscape.service] == right[dt.smartscape.service]}
| fieldsAdd this_total = arraySum(this_week)
| fieldsAdd last_total = arraySum(right.last_week)
| filter isNotNull(this_total) and isNotNull(last_total) and last_total > 0
| fieldsAdd wow_change_pct = round((this_total - last_total) / last_total * 100, decimals: 1)
| sort wow_change_pct desc
| limit 20
| fields dt.smartscape.service, this_total, last_total, wow_change_pct
```

### Growth Rate Detection (30-Day Trend)

```dql
timeseries cpu = avg(dt.host.cpu.usage), from: now()-30d, interval: 1d,
  by: {dt.smartscape.host}
| fieldsAdd current = arrayLast(cpu)
| fieldsAdd baseline = arrayFirst(cpu)
| filter isNotNull(current) and isNotNull(baseline)
| fieldsAdd daily_growth = (current - baseline) / 30
| fieldsAdd total_growth_pct = round((current - baseline) / baseline * 100, decimals: 1)
| sort daily_growth desc
| limit 20
| fields dt.smartscape.host, baseline, current, daily_growth, total_growth_pct
```

### Acceleration Detection

Detect whether the rate of change is itself increasing:

```dql
timeseries cpu = avg(dt.host.cpu.usage), from: now()-14d, interval: 1d,
  by: {dt.smartscape.host}
| fieldsAdd diffs = arrayDiff(cpu)
| fieldsAdd avg_acceleration = arrayAvg(diffs)
| fieldsAdd current = arrayLast(cpu)
| filter isNotNull(current) and avg_acceleration > 0
| sort avg_acceleration desc
| limit 20
| fields dt.smartscape.host, current, avg_acceleration
```

### Service Error Rate Trend

```dql
timeseries {
  errors = sum(dt.service.request.failure_count),
  total = sum(dt.service.request.count)
}, from: now()-7d, interval: 1h, by: {dt.smartscape.service}
| fieldsAdd error_rate_now = arrayLast(errors) / arrayLast(total) * 100
| fieldsAdd error_rate_7d_avg = arrayAvg(errors) / arrayAvg(total) * 100
| filter isNotNull(error_rate_now) and error_rate_7d_avg > 0
| fieldsAdd rate_change = error_rate_now - error_rate_7d_avg
| sort rate_change desc
| limit 20
| fields dt.smartscape.service, error_rate_now, error_rate_7d_avg, rate_change
```

### Service Response Time Trend (Week-over-Week)

```dql
timeseries rt_now = avg(dt.service.request.response_time),
  from: now()-7d, interval: 1h, by: {dt.smartscape.service}
| join [
    timeseries rt_prev = avg(dt.service.request.response_time),
      from: now()-14d, to: now()-7d, interval: 1h,
      by: {dt.smartscape.service}
  ], on: {left[dt.smartscape.service] == right[dt.smartscape.service]}
| fieldsAdd avg_now = arrayAvg(rt_now) / 1000
| fieldsAdd avg_prev = arrayAvg(right.rt_prev) / 1000
| filter isNotNull(avg_now) and avg_prev > 0
| fieldsAdd wow_change_pct = round((avg_now - avg_prev) / avg_prev * 100, decimals: 1)
| sort wow_change_pct desc
| limit 20
| fields dt.smartscape.service, avg_now, avg_prev, wow_change_pct
```

---

## Trend Result Presentation

| Column | Content |
|--------|---------|
| Rank | 🥇 🥈 🥉 by growth rate (descending) |
| Service / Host | Entity |
| Baseline | Value at start of window |
| Current | Latest value |
| Growth/Day | Daily rate of change |
| WoW Change | Week-over-week % change |
| Action | ✅ Stable / ⚠️ Monitor / 🔴 Escalate |

---

## Best Practices

- Use `timeseries-novelty-detection` first to confirm a trend is structural,
  not just noise; `filterSpikes: true` removes transient spikes before
  trend analysis
- Use `CHANGE_IN_VALUES` for step changes (permanent baseline shifts);
  use `TREND_IN_VALUES` for gradual directional movement
- Week-over-week joins require identical `interval:` values on both sides
  so join keys align; right-side fields are prefixed with `right.`
- `dt.service.request.response_time` is in nanoseconds — divide by
  `1000000` for ms
- A positive acceleration (`arrayAvg(arrayDiff(arr)) > 0`) means the metric
  is speeding up — worth flagging even if the current value is acceptable